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1.0  SUMMARY 

Everyone  agrees  that  intelligence  analysis  is  an  intensely  cognitive  process  and  that  the  quality  of 
an  analyst’s  cognition  determines  the  quality  of  his  or  her  analysis.  In  the  past  ten  years,  the 
intelligence  community  (IC)  has  increasingly  used  the  term  critical  thinking  to  describe  the 
cognitive  aspects  of  analysis.  Critical  thinking  has  appeared  in  the  IC’s  competency  lists, 
governing  documents,  curricula,  and  training  materials.  It  is  described  as  an  important 
contributor  to  effective  analysis,  and  its  lack  has  been  identified  as  a  contributor  to  analytic 
failures. 

I  conducted  a  research  synthesis  to  gain  a  better  understanding  of  critical  thinking  within  and 
beyond  the  field  of  intelligence  analysis  and  discovered  that  it  is  an  ambiguous  concept.  I 
reviewed  over  500  documents  that  addressed  critical  thinking,  including  128  documents  from  the 
intelligence  literature.  Critical  thinking  is  universally  described  as  important,  yet  it  isn’t  clear 
that  all  authors  mean  the  same  thing  when  they  write  about  critical  thinking.  Critical  thinking 
appears  to  be  embraced  as  a  concept  without  any  interpretation,  evaluation,  or  analysis  to 
confirm  that  those  talking  about  critical  thinking  all  mean  the  same  thing.  I  found: 

•  Little  agreement  on  what  critical  thinking  is,  with  multiple  definitions  of  critical 
thinking  in  the  literature  of  both  intelligence  analysis  and  other  fields. 

•  Wide  variation  in  the  description  of  the  thinking  skills  and  dispositions  required  to 
think  critically  across  all  fields. 

•  Training  courses  for  critical  thinking,  in  all  fields,  with  different  objectives,  content, 
teaching  approaches,  and  opportunities  for  practice. 

•  Customized  and  standardized  assessment  tools  that  are  tailored  to  different  theories 
of  critical  thinking. 

•  Conflicting  descriptions  of  how  to  apply  critical  thinking  to  the  process  of 
intelligence  analysis. 

Although  critical  thinking  is  believed  to  improve  analysis,  and,  intuitively  speaking,  who  would 
think  otherwise,  I  found  no  reports  describing  how  critical  thinking  actually  improves  the 
analytic  result.  The  lack  of  agreement  about  what  critical  thinking  is  and  how  to  apply  it  limits 
any  demonstration  of  whether  or  not  it  will  provide  value.  Without  evidence  identifying 
specifically  what  aspects  of  critical  thinking  actually  improve  analysis,  current  investments  in 
advancing  critical  thinking  throughout  the  intelligence  community  may  not  result  in  the 
anticipated  benefit. 

This  report  suggests  a  number  of  opportunities  to  validate  the  concept  of  critical  thinking  more 
rigorously  and  to  synthesize  and  develop  the  methodologies  necessary  to  make  critical  thinking  a 
viable  part  of  the  work  of  the  IC. 
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2.0  INTRODUCTION 

The  Framework  for  Analytic  Cognition  highlighted  the  challenges  facing  the  US  IC: 

Starting  with  Pearl  Harbor,  the  US  IC  has  faced  criticism  for  failing  to  predict  or  warn  of 
future  events.  Numerous  examples  of  these  failures  have  been  cited,  including  two  of  the 
most  recent  and  highly  publicized  examples:  failure  to  warn  against  the  9-11  attack  and 
incorrectly  estimating  the  size  and  substance  of  Iraq’s  weapons  program.  As  a  result, 
numerous  groups  and  commissions  have  been  established  since  the  end  of  World  War  II 
to  study  “what  went  wrong.” 

All  of  these  efforts  have  resulted  in  proposals  to  “fix”  the  IC  and  thereby  improve  the 
quality  of  its  products.  While  the  proposals  have  come  from  different  groups,  a  certain 
commonality  exists  between  the  proposals.  As  could  be  expected,  an  increase  in 
resources  has  been  proposed.  However,  the  belief  persists  among  both  critics  and 
supporters  of  the  IC  that  the  solution  does  not  lie  solely  in  providing  more  resources. 
Instead,  an  intelligence  product  can  only  be  improved  by  obtaining  improvements  in  both 
the  quality  and  productivity  of  analysts.  In  this  context,  the  solutions  proposed,  past  and 
present,  have  included  selecting  more  qualified  personnel,  enhancing  training,  developing 
new  methodologies  for  analysis,  and  organizing  analytical  efforts.1 2 

Beginning  in  the  late  1990s,  the  specific  topic  of  critical  thinking  has  been  increasingly 
mentioned  as  one  of  the  “fixes”  to  improve  the  quality  of  analysis.  James  Asker  was  one  of  the 
earliest  authors  to  highlight  critical  thinking: 

Grumbling  lawmakers  say  the  intelligence  failure  to  detect  India's  nuclear  test 
preparations  was  rooted  in  defects  flagged  long  ago,  but  continually  sidestepped  by  both 
Congress  and  spy  agencies.  A  quick  review  led  by  Adm.  David  Jeremiah,  a  former  vice 
chairman  of  the  Joint  Chiefs  of  Staff,  produced  a  litany  of  familiar  flaws:  lack  of  critical 
thinking  and  analytic  rigor,  weak  internal  coordination,  botched  collection  priorities,  and 

9 

too  much  reliance  on  technical  intelligence-gathering,  too  little  on  spies. 

Stephen  Marrin,  bringing  critical  thinking  more  explicitly  to  the  attention  of  the  intelligence 
community,  included  this  quote  in  several  of  his  articles  published  from  2002-20043  about 
improving  intelligence  analysis.  Marrin  was  one  of  the  first  to  emphasize  the  need  for  critical 
thinking. 

Stephane  Lefebvre  also  highlighted  the  importance  of  critical  thinking  in  2003: 

If  we  accept  the  cognitive  model  developed  by  Operating  Systems,  Inc.  as  valid,  then  it 
becomes  clear  why  analysts  must  be  trained  and  encouraged  to  develop  the  necessary 
skills  to  think  critically  and  innovatively.  Objectivity  is,  as  Michael  Herman  argues,  an 


1  Kampman,  C.,  Mangio,  C.,  Parry,  T.,  and  Wilkinson,  B.  Framework  for  Analytic  Cognition  (FAC):  A  Guide  for 
Doing  All-Source  Intelligence  Analysis,  Air  Force  Research  Laboratory,  Wright-Patterson  Air  Force  Base,  OF1,  [in 
press],  p.  2. 

2  Anonymous.  "Same  01',  Same  01',"  Aviation  Week  and  Space  Technology,  Vol.  148,  No.  23,  1998,  pp.  21. 

3  See  Marrin,  S.  "CIA's  Kent  School:  A  Step  in  the  Right  Direction,"  International  Studies  Association  Conference, 
New  Orleans,  LA,  March  18,  2002;  Marrin,  S.  "Improving  CIA  Analysis  by  Overcoming  Institutional  Obstacles," 
in  Bringing  Intelligence  About:  Practitioners  Reflect  on  Best  Practices,  ed.  R.G.  Swenson,  Joint  Military 
Intelligence  College,  Washington,  D.C.,  2003,  pp.  41-59;  and  Marrin,  S.  "CIA's  Kent  School:  Improving  Training 
for  New  Analysts,"  International  Journal  of  Intelligence  and  Counterintelligence,  Vol.  16,  No.  4,  2003-2004,  pp. 
609-637. 
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‘elusive  ideal.’  This  is  important  if  intelligence  analysts  are  to  produce  estimates  that  are 
as  unbiased  and  free  of  logical  fallacies  as  possible.  To  wit,  if  anything  should  be  taken 
for  granted  it  is  the  centrality  of  critical  reflection,  or  boundary  exploration  and  critique, 
to  all  forms  of  analysis.  It  is  perhaps  a  poor  reflection  on  the  current  analytical  culture 
that  critical  thinking  as  an  activity  has  to  be  made  explicit.59  4 

Major  George  E.  Lewis  III  also  wrote  about  how  the  lack  of  critical  thinking  contributed  to 
intelligence  failures  in  a  document  published  in  2005: 

Short  of  a  lack  of  information,  failures  in  thinking  are  perhaps  the  number  one  reason 
behind  intelligence  failures.  Failures  or  the  inability  to  recognize  bias,  understand 
cultural  deviations,  and  ignoring  alternative  premises  are  only  a  few  of  the  traps  that  an 
intelligence  analyst  can  fall  prey  to  if  he/she  cannot  think  critically.  Of  the  eight  critical 
skills  within  the  core  competencies  of  intelligence  analysts,  critical  thinking  is  the  most 
crucial  and  is  often  the  least  trained.5 

In  the  years  following  the  identification  of  the  importance  of  critical  thinking,  it  has  appeared  on 
lists  of  analyst  competencies,  is  increasingly  included  in  analyst  training  curricula,  and  is  now 
mentioned  explicitly  in  governing  documents  such  as  The  US  Intelligence  Community's  Strategic 
Human  Capital  Plan.6  Yet  it  is  not  always  clear  what  anyone  means  when  the  term  critical 
thinking  is  used  nor  how  to  develop  the  competency  in  analysts  in  order  to  improve  the  analysis 
process  and  reduce  failures  caused  by  “a  lack  of  critical  thinking.” 

Noel  Hendrickson  noted  the  ongoing  lack  of  understanding  of  critical  thinking  in  intelligence 
analysis  in  2008: 

Critical  thinking  appears  on  almost  every  list  of  the  essential  skills  for  intelligence 
analysts.1  But  any  corresponding  attempt  to  define  critical  thinking  more  precisely  is 
seldom  encountered.  And,  on  those  rare  occasions  when  definitions  are  offered,  they 
inevitably  (albeit  quite  understandably)  amount  to  new  applications  of  existing 
approaches. 2  Yet,  these  existing  approaches  derive  from  more  general  academic  attempts 
to  create  better  thinkers  and  not  from  any  specific  concern  for  the  problems  of 
intelligence  analysis.  Thus,  despite  the  undeniable  importance  of  critical  thinking  in 
intelligence  analysis,  few  if  any  attempts  are  made  to  define  critical  thinking  specifically 
for  intelligence.7 

In  support  of  Hendrickson’s  statement,  I  found  a  total  of  21  intelligence  analysis  documents  that 
contained  definitions  for  critical  thinking.  Seventeen  of  the  items  had  been  published  during  or 
before  2008.  Seven  of  the  documents  referenced  existing  scholarly  definitions,  13  of  the 
documents  provided  new,  unique  definitions,  and  some  did  both.  No  reports  provided 
information  on  how  critical  thinking  improves  intelligence  analysis  nor  gave  any  insight  into 
which  version  of  critical  thinking  might  be  the  most  useful  for  intelligence  analysis. 

4  Lefebvre,  S.J.  "Strategic  Warning:  If  Surprise  is  Inevitable,  What  Role  for  Analysis?  POSTER  Presentation 
TC99,"  International  Studies  Association  (ISA)  44th  Annual  International  Convention,  Portland,  OR,  February  25  - 
March  1,2003,  pp.  23-24. 

5  Lewis,  G.E.  Army  Intelligence  Analysis:  Transforming  Army  Intelligence  Analysis  Training  and  Doctrine  to  Serve 
the  Reasonable  Expectations  and  Needs  of  Echelons  Corps  and  Below  Commanders,  Consumers,  and  Customers, 
ADA435988,  United  States  Army  Command  and  General  Staff  College,  Ft.  Leavenworth,  KS,  2005,  p.  32. 

6  Anonymous.  The  US  Intelligence  Community's  Strategic  Human  Capital  Plan:  An  Annex  to  the  US  National 
Intelligence  Strategy ;,  Office  of  the  Director  of  National  Intelligence,  Washington,  D.C.,  2006. 

7  Hendrickson,  N.  "Critical  Thinking  in  Intelligence  Analysis,"  International  Journal  of  Intelligence  and 
Counterintelligence,  Vol.  21,  No.  4,  2008,  p.  679. 
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Critical  thinking’s  multiple  definitions,  the  varied  descriptions  of  how  to  apply  critical  thinking 
to  intelligence  analysis,  and  the  lack  of  information  demonstrating  the  benefit  of  critical  thinking 
illustrate  the  need  for  a  better  understanding  of  critical  thinking  and  how  it  might  support 
effective  intelligence  analysis.  While  we  can  intuitively  accept  that  improvements  in  cognitive 
processes  described  by  the  term  critical  thinking  can  improve  analysis,  it  would  be  beneficial  to 
reach  agreement  on  specifically  what  leads  to  the  desired  cognitive  improvements,  including 
such  topics  as  thinking  skills,  thinking  dispositions,  and  effective  training. 

2.1  Purpose 

The  purpose  of  this  research  synthesis  is  to  provide  an  understanding  of  the  concept  of  critical 
thinking,  how  the  concept  is  represented  in  the  intelligence  literature,  and  its  potential 
applicability  to  intelligence  analysis.  We  hoped  to  learn: 

•  What  is  critical  thinking? 

•  What  is  the  state  of  knowledge  regarding  critical  thinking? 

•  Can  we  train  someone  to  think  critically? 

•  Can  we  assess  someone’s  ability  to  think  critically? 

•  How  can  this  understanding  of  critical  thinking  be  applied  to  improving  intelligence 
analysis? 
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3.0  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 

This  project  builds  upon  work  completed  in  the  State  of  Knowledge  Relative  to  Intelligence 

o 

Analysis  project.  The  initial  documents  searched  were  those  collected  for  the  state  of 
knowledge  project.  As  a  result  of  the  close  relationship  between  projects,  the  critical  thinking 
project  followed  many  of  the  same  practices  used  in  the  state  of  knowledge  project  including: 

•  Use  of  the  RefWorks  Bibliographic  Management  System  as  the  database  for  the 
project  bibliography. 

•  Search  in  a  wide-range  of  topics  and  professions  beyond  intelligence  analysis. 

•  Use  of  a  two-level  determination  of  document  relevance  and  usefulness  for  new 
documents: 

o  Level  1  to  occur  during  active  searching,  providing  an  initial  detennination 
of  relevance.  Those  items  determined  to  be  relevant  at  this  stage  were  saved 
for  later  review.  All  saved  documents  were  entered  into  RefWorks. 

o  Level  2  to  occur  during  the  document  review  process  to  detennine 

usefulness,  with  keyword  descriptors  added  to  RefW orks  to  identify  the 
specific  kind  of  information  contained  in  the  document.  “Useful”  was  based 
on  whether  the  document  would  add  new  knowledge  or  a  new  perspective. 
(For  example,  we  did  not  save  every  logic  textbook  or  every  book  or  article 
describing  a  theoretical  approach  to  teaching  critical  thinking.) 

•  Integration  of  new  documents  into  the  existing  intelligence  analysis  repository 
rather  than  creating  a  separate  critical  thinking  repository  with  duplicate  documents. 
All  critical  thinking  documents  are  assigned  the  keyword  CT  as  a  descriptor. 

•  Reread  all  documents  a  second  time  after  developing  a  baseline  of  understanding 
from  the  initial  review. 

3.1  Information  Acquisition 

In  addition  to  the  field  of  intelligence  analysis,  modem  research  and  reflection  on  critical 
thinking  encompasses  multiple  fields  of  study,  including  philosophy,  psychology,  education, 
communications,  accounting,  and  medicine.  The  search  conducted  for  this  study  resulted  in 
documents  addressing  critical  thinking  concepts  from  each  of  the  fields  of  study  named,  as  well 
as  information  on  assessments,  training,  guidelines,  application,  and  research  results. 

After  securing  a  sufficient  number  of  documents  to  provide  a  working  understanding  of  critical 
thinking,  I  focused  on  identifying: 

•  Early  20th  Century  documents  to  understand  the  genesis  of  the  term  critical  thinking. 

•  Recently  published  documents  to  gain  the  most  up-to-date  theory  and  knowledge 
regarding  critical  thinking. 

•  Documents  specifically  focused  on  critical  thinking  in  intelligence  analysis. 

•  Documents  that  described  lessons  learned  from  studies  in  various  fields  about  the 
efficacy  of  teaching  or  assessing  critical  thinking. 

The  steps  I  used  to  identify  potentially  relevant  documents  were: 


8  State  of  Knowledge  Relative  to  Intelligence  Analysis,  Contract  Number  FA8650-05-C-6556,  Air  Force  Research 
Laboratory,  Wright-Patterson  Air  Force  Base,  OFF  Contract  conducted  January  2007-  December  2010. 
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1 .  Search  all  documents  in  the  intelligence  analysis  repository  for  the  term  critical  thinking. 

2.  For  those  documents  identified  as  pertaining  to  critical  thinking,  search  the  individual 
document  to  find  what  was  said. 

3.  Review  reference  lists  in  documents  pertaining  to  critical  thinking  to  identify  additional 
documents  to  review  and  evaluate.  Searching  the  references  provided  a  very  rich  source  of 
information. 

4.  Use  Google  search  to  identify  additional  documents,  articles,  and  resources.  Search  terms 
included:  “critical  thinking,”  “critical  thinking  assessment,”  “critical  thinking  training,” 
“critical  thinking  and  intelligence  analysis,”  “critical  reasoning,”  author  names,  article  and 
document  names  found  in  references  within  documents,  and  tools  or  resources  mentioned  in 
documents. 

5.  Follow  interesting  leads  to  other  relevant  journals,  web  sites,  new  documents,  new  authors, 
and  new  tools. 

The  search  was  careful,  but  it  was  not  exhaustive.  For  example,  a  search  of  “critical  thinking”  on 
Google  returned  417,000  results.  A  search  of  Amazon.com  for  “critical  thinking”  returned  a 
possible  4,708  books,  while  a  search  of  the  term  “thinking  critically”  returned  a  more  reasonable 
number  of  190  books.  When  faced  with  this  volume  of  results,  I  reviewed  the  initial  pages  of 
retrieved  items  to  determine  relevance  then  moved  on  to  new  sources  or  search  terms  when  the 
pages  began  to  contain  items  of  little  relevance. 

3.2  Documents  in  the  Repository 

The  documents  reviewed  for  relevance  to  critical  thinking  represent  not  only  a  variety  of  fields, 
but  also  were  written  to  serve  a  wide  variety  of  purposes.  The  majority  of  documents  did  not 
describe  or  report  on  experiments  and  studies  about  critical  thinking.9  Rather,  the  majority  of  the 
documents  fit  into  the  following  categories: 

•  Discussions  of  theories  about  thinking,  critical  thinking,  or  related  topics  such  as 
analogy  or  deductive  thinking.  Typically  these  theories  are  presented  as  fact. 

•  Textbooks  proposing  how  to  think  critically  or  how  to  teach  thinking. 

•  Publications  that  proposed  methods  for  applying  critical  thinking. 

•  Results  of  academic  research  that  contribute  new  perspectives  or  knowledge  on 
critical  thinking  or  a  related  topic. 

•  Course  descriptions  or  syllabi  that  included  critical  thinking  as  a  primary  focus  or  as 
one  topic  in  the  course. 

•  Documents  about  some  other  topic  such  as  intelligence  analysis  that  mention 
critical  thinking  briefly  or  discussed  critical  thinking  with  respect  to  the  primary 
topic. 

Those  documents  that  reported  on  experiments  or  studies  fell  into  multiple  categories  as  well: 


9  The  difficulty  in  finding  empirical  studies  of  critical  thinking  was  also  noted  by  Claudia  Maria  Alvarez  Ortiz  in  her 
Master’s  Thesis  (Ortiz,  C.M.A.  Does  Philosophy  Improve  Critical  Thinking  Skills?  MA  thesis,  University  of 
Melbourne,  2007). 
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•  True  experiments  testing  a  theory  with  a  well-designed  study  including  control 
groups  or  comparisons  to  groups  that  did  not  participate  in  the  experiment  such  as 
Glaser’s  An  Experiment  in  the  Development  of  Critical  Thinking}0 

•  Summaries  of  reports  on  widely  varying  initiatives  such  as  Teaching  and 
Measuring  Critical  Thinking, 1 1  which  summarized  the  results  of  attempts  to  include 
critical  thinking  in  nursing  education. 

•  Studies  of  a  single  tool  designed  to  enhance  one  aspect  of  critical  thinking  such  as 
the  report  on  the  use  of  an  automated  argument  analysis  tool  in  Argument  Maps 
Improve  Critical  Thinking.12 

•  Reports  of  long  term  studies  to  develop  and  test  a  theory  such  as  King  and 

1 3 

Kitchener’s  Reflective  Judgment  Model. 

•  Meta-analyses  of  existing  empirical  studies  on  the  results  of  various  approaches  to 
teaching  critical  thinking  such  as  Does  Philosophv  Improve  Critical  Thinking 
Skill  si14 

3.3  This  Report 

This  report  evaluates  and  summarizes  the  results  of  the  research  described  above,  with  an 
emphasis  on  how  critical  thinking  might  apply  to  intelligence  analysis. 

The  volume  of  writing  about  critical  thinking  precludes  a  comprehensive  summary  in  this  report 
of  all  of  the  critical  thinking  documents  in  the  repository.  I  selected  publications  for  inclusion 
that  illustrate  the  history  of  the  tenn,  reflect  the  current  state  of  knowledge,  demonstrate  a 
breadth  of  understanding  across  disciplines,  describe  lessons  learned,  or  pertain  to  intelligence 
analysis. 

The  remainder  of  the  report  is  organized  as  follows: 

•  Results  and  discussion 

o  What  is  critical  thinking? 

o  Enablers  of  critical  thinking:  skills,  dispositions,  and  personal  epistemology 
o  Can  we  train  someone  to  think  critically? 
o  Assessing  critical  thinking 

o  Critical  thinking  in  the  intelligence  analysis  literature 

•  Conclusions 

•  Appendix  A  -  Critical  Thinking  Definitions 

10  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941. 

11  Jones,  E.A.,  Hoffman,  S.,  Moore,  L.M.,  Ratcliff,  G.,  Tibbetts,  S.  &  Click,  B.A.L.  National  Assessment  of  College 
Student  Learning:  Identifying  College  Graduates'  Essential  Skills  in  Writing,  Speech  and  Listening,  and  Critical 
Thinking.  Final  Project  Report,  National  Center  on  Post  Secondary  Teaching,  Learning,  and  Assessment,  University 
Park,  PA,  1995. 

12  Twardy,  C.R.  Argument  Maps  Improve  Critical  Thinking,  Monash  University,  Melbourne,  Victoria,  Australia, 
[undated]. 

13  See  King,  P.M.  &  Kitchener,  K.S.  Developing  Reflective  Judgment:  Understanding  and  Promoting  Intellectual 
Growth  and  Critical  Thinking  in  Adolescents  and  Adults,  Jossey-Bass  Publishers,  San  Francisco,  CA,  1994  and 
King,  P.M.  &  Kitchener,  K.S.  "The  Reflective  Judgment  Model:  Twenty  Years  of  Research  on  Epistemic 
Cognition,"  in  Personal  Epistemology’:  The  Psychology’  of  Beliefs  about  Knowledge  and  Knowing,  eds.  B.K.  Hofer 
&  P.R.  Pintrich,  L.  Erlbaum  Associates,  Mahwah,  NJ,  2002,  pp.  37-61. 

14  Ortiz,  C.M.A.  Does  Philosophy  Improve  Critical  Thinking  Skills?  MA  Thesis,  University  of  Melbourne,  2007. 
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4.0  RESULTS  AND  DISCUSSION 


4.1  What  is  Critical  Thinking? 

Little  agreement  exists  about  what  critical  thinking  is,  as  illustrated  by  the  fact  that  I  found  118 
unique  definitions  of  critical  thinking.  This  lack  of  agreement  is  due  primarily  to  the  fact  that 
critical  thinking  is  only  a  theoretical  concept.  The  term  is  typically  used  to  describe  a  quality  of 
thinking  expected  to  result  in  high  quality  outcomes.  However,  since  critical  thinking  is  a 
theoretical  concept,  it  can’t  be  touched,  can’t  be  weighed  or  measured  physically,  and  can’t  be 
easily  recognized  when  someone  is  engaged  in  it.  The  concept  is  understood  and  described 
through  the  lens  of  the  education  and  experiences  of  the  person  creating  the  description. 

A  method  to  prove  or  disprove  any  of  the  definitions  does  not  exist.  The  definitions  can  only  be 
accepted  as  useful  or  rejected  as  inadequate  by  readers  based  on  their  own  education  and 
experiences,  on  their  familiarity  with  the  specific  tenns  used,  and  on  the  intended  application  of 
the  concept  of  critical  thinking. 

Some  of  the  lenses  through  which  critical  thinking  can  be  understood  include: 

•  Philosophy:  Philosophers  believe  that  critical  thinking  originated  with  the  earliest 
Greek  philosophers,  and  it  remains  a  key  element  of  philosophy  and  logic  today. 

•  Education:  Educators  are  interested  in  ensuring  the  K- 12  educational  system 
adequately  addresses  critical  thinking  for  all  students.  They  are  interested  in  how  to 
teach  critical  thinking  and  what  the  outcomes  of  critical  thinking  might  be. 

•  Psychology:  Cognitive  psychologists  focus  on  the  cognition  associated  with  critical 
thinking,  that  is,  what  is  going  on  in  the  brain  while  critical  thinking  is  taking  place. 

•  Communications:  Communications  specialists  focus  on  understanding  and 
persuading  in  written  and  oral  communications. 

•  Medicine:  The  medical  community,  especially  the  nursing  profession,  wants  to 
ensure  that  medical  practitioners  have  critical  thinking  capabilities  to  support  their 
clinical  work. 

•  Law:  The  legal  profession  has  long  believed  that  critical  thinking  ability  is  an 
important  success  factor  in  law  school  and  in  the  practice  of  law. 

•  Intelligence:  More  recently,  the  intelligence  community  has  identified  critical 
thinking  as  a  success  factor  for  effective  intelligence  analysis. 

Robert  Sternberg  provided  an  overview  of  the  three  primary  academic  fields  where  research  and 
theory  development  regarding  critical  thinking  provided  the  initial  foundations  for  the  concept: 

The  study  of  critical  thinking  is  of  particular  interest  because  of  its  confluence  of  three 
traditions  of  thought— the  educational,  the  philosophical,  and  the  psychological.  Indeed, 
if  there  is  a  modern-day  founder  of  the  "critical-thinking  movement,"  it  is  almost 
certainly  John  Dewey,  who  was  simultaneously  an  educator,  a  philosopher,  and  a 
psychologist.15 

The  Philosophical  Tradition.  The  concern  of  philosophers  with  the  elements  of  critical 
thinking  dates  back  to  Ancient  time.  If  Dewey  is  the  modern-day  founder  of  the  critical- 


15  Sternberg,  R.J.  Critical  Thinking:  Its  Nature,  Measurement,  and  Improvement,  Yale  University,  New  Haven,  CT, 
1986,  p.4. 
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thinking  movement,  then  Plato  and  Aristotle  would  be  its  ancient  founders.  In  more  recent 
times,  philosophers  such  as  Ennis  (in  press),  Lipman  (in  press),  and  Paul  (in  press)  have 
devoted  their  attention  to  understanding  the  bases  of  critical  thinking.  Philosophers  have 
focused  their  attention  not  so  much  upon  the  requirements  of  critical  thinking  in  the 
classroom,  but  upon  the  requirements  of  formal  logical  systems.16 

The  Psychological  Tradition.  Psychologists  interested  in  the  nature  of  critical  thinking, 
such  as  Bransford  (1984),  Bruner  (1960,  1961),  Feuerstein  (1980),  and  Sternberg  (1985), 
have  been  particularly  concerned  with  characterizing  critical  thinking  as  it  is  performed 
under  the  limitations  of  the  person  and  the  environment.17 

The  Educational  Tradition.  In  the  educational  tradition  of  theorizing  are  leading  figures 

such  as  Bloom  (1955),  Gagne  (1965),  Perkins  (1981),  and  Renzulli  (1976),  whose 

theorizing  seems  directly  responsive  to  the  skills  needed  by  children  in  the  classroom  for 

problem  solving,  decision  making,  and  concept  learning.  Bloom’s  (1956)  famous 

taxonomy  of  cognitive  skills  and  Gagne's  (1965)  well-known  hierarchy  of  learning  skills 

1  8 

have  seen  widespread  application  in  classroom  situations  and  even  textbook  creation. 

I  will  attempt  to  answer  the  question  “What  is  critical  thinking?”  by  examining  several 
perspectives: 

•  The  history  and  origin  of  the  term  critical  thinking 

•  The  many  definitions  of  critical  thinking 

•  The  relationship  of  critical  thinking  to  formal  logic 

Finally,  this  section  concludes  with  a  description  of  critical  thinking  synthesized  from  the 
research  findings. 

4.1.1.  The  Origin  of  the  Term  Critical  Thinking 

The  true  origin  of  the  term  critical  thinking  is  not  definitively  known.  Several  scholars  provided 
suggestions,  but  my  research  found  earlier  examples  of  the  term  than  those  mentioned.  Several 
scholars  also  mentioned  early  influencers  of  the  concept  of  critical  thinking  such  as  William 
Hare  who  wrote: 

The  current  obsession  has  created  the  impression  that  we  have  suddenly,  and  at  long  last, 
seen  through  the  deficiencies  of  traditional  education  which  have  blinded  us  for  so  long 
to  the  insight  we  have  now  achieved.  We  have  stumbled  out  of  the  cave  into  the  daylight. 
If,  however,  critical  thinking  really  is  a  central  aim  of  education,  it  would  be  remarkable 
indeed  if  this  were  a  discovery  of  the  late  twentieth  century;  having  somehow  eluded 
philosophers  for  more  than  2000  years. 

In  fact  the  history  of  this  ideal  can  be  traced  back  through  philosophy  to  the  earliest 
times.  That  account  would  include  reference  to  Mill  on  keeping  one’s  mind  open  to 
criticism,  Kant  on  thinking  for  oneself,  Descartes  on  the  need  to  assess  (not  simply  be 
acquainted  with)  the  views  of  other  philosophers,  and  on  through  the  history  of 
philosophy  to  its  origins,  in  the  western  tradition  at  least,  in  the  Socratic  emphasis  on  the 
examined  life.i  These  ideas  are  central  to  any  account  of  the  intellectual  virtues, 


16  Sternberg,  p.  4. 

17  Sternberg,  p.  5. 

1S  Sternberg,  p.  6. 

10 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-2014-1598,  Cleared  11  April  2014 


including  wisdom,  judgment,  and  open-mindedness,  and  it  is  within  this  family  of 
concepts  that  critical  thinking  emerges  as  an  ideal.19 

The  concept  of  high  quality  thinking,  described  with  many  different  names,  was  popular  in  the 
first  half  of  the  20th  century.  Although  the  term  critical  thinking  appeared  in  publications,  it  was 
initially  just  one  of  many  terms  used  synonymously  for  good  thinking.  A  list  of  the  terms  used 
to  describe  high  quality  thinking  and  some  samples  of  where  the  terms  were  used  include: 

90 

•  Reflective  thinking  (John  Dewey,  How  We  Think  ) 

2 1 

•  Critical  thinking  (Julius  Boraas,  Teaching  To  Think,'  and  Max  Black,  Critical 
Thinking22) 

23 

•  Straight  thinking  (E.  L.  Clarke,  The  Art  of  Straight  Thinking “  and  Robert  Thouless, 
Straight  and  Crooked  Thinking  4) 

•  Effective  thinking  (J.  Jastrow,  Effective  Thinking25) 

26 

•  Clear  thinking  (R.  W.  Jepson,  How  to  Think  Clearly •  ) 

•  Correct  thinking  (C.  H.  Patterson,  Principles  of  Correct  Thinking21) 

The  earliest  use  I  found  of  the  specific  term  critical  thinking  was  in  a  book  titled  Thinking  and 
Learning  to  Think,  published  in  1901.  This  book  also  used  multiple  terms  to  describe  good 
thinking,  including  clear  thinking,  distinct  thinking,  adequate  thinking,  and  exhaustive  thinking. 

28 

No  one  has  done  more  to  shape  the  critical  thinking  of  the  world  than  Kant; 

Although  multiple  authors  identified  John  Dewey  as  the  father  of  modern  critical  thinking,  his 
preferred  term  was  reflective  thinking  rather  than  critical  thinking.  Dewey’s  definition  of 
reflective  thinking  is: 

Active,  persistent,  and  careful  consideration  of  any  belief  or  supposed  form  of  knowledge 
in  the  light  of  the  grounds  that  support  it  and  the  further  conclusions  to  which  it  tends 
constitutes  reflective  thought.29 

Dewey  also  described  the  qualitative  aspects  of  thought  that  contribute  to  successful  reflective 
thinking  -  the  importance  of  being  critical  when  thinking: 

There  may,  however,  be  a  state  of  perplexity  and  also  previous  experience  out  of  which 
suggestions  emerge,  and  yet  thinking  need  not  be  reflective.  For  the  person  may  not  be 


19  Hare,  W.  "Critical  Thinking  as  an  Aim  of  Education,"  in  Aims  of  Education,  ed.  R.  Marples,  Routledge,  London, 
UK,  1999,  p.  86. 

20  See  Dewey,  J.  How  We  Think,  D.  C.  Heath  &  Co.,  Chicago,  IL,  1910  and  Dewey,  J.  How  We  Think:  A 
Restatement  of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed.,  Regnery,  Chicago,  IL,  1933. 

21  Boraas,  J.  Teaching  to  Think,  Macmillan,  New  York,  NY,  1922. 

22  Black,  M.  Critical  Thinking:  An  Introduction  to  Logic  and  Scientific  Method,  2nd  ed.,  Prentice-Hall,  New  York, 
NY,  1952. 

23  Clarke,  E.L.  The  Art  of  Straight  Thinking,  D.  Appleton  and  Company,  New  York,  NY,  1929. 

24  Thouless,  R.H.  Straight  and  Crooked  Thinking,  Pan  Books  Ltd.,  London,  UK,  1953. 

25  Jastrow,  J.  Effective  Thinking,  World  Publishing,  Cleveland,  OH,  1931. 

26  Jepson,  R.W.  How  to  Think  Clearly,  Longmans,  Green  and  Company,  New  York,  NY,  1936. 

27  Patterson,  C.H.  Principles  of  Correct  Thinking,  Longmans,  Green  and  Co.,  New  York,  NY,  1937. 

28  Schaeffer,  N.C.  Thinking  and  Learning  to  Think,  L.  B.  Lippincott  Co.,  Philadelphia  PA,  1901,  p.  227. 

29  Dewey,  J.  How  We  Think,  D.  C.  Heath  &  Co.,  Chicago,  IL,  1910,  p.  6  and  Dewey,  J.  How  We  Think:  A 
Restatement  of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed.,  Regnery,  Chicago,  IL,  1933,  p. 
9. 
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sufficiently  critical  about  the  ideas  that  occur  to  him.  He  may  jump  at  a  conclusion 
without  weighing  the  grounds  on  which  it  rests;  he  may  forego  or  unduly  shorten  the  act 
of  hunting,  inquiring;  he  may  take  the  first  'answer,'  or  solution,  that  comes  to  him 
because  of  mental  sloth,  torpor,  impatience  to  get  something  settled.  One  can  think 
reflectively  only  when  one  is  willing  to  endure  suspense  and  to  undergo  the  trouble  of 
searching.  To  many  persons  both  suspense  of  judgment  and  intellectual  search  are 
disagreeable;  they  want  to  get  them  ended  as  soon  as  possible.  They  cultivate  an  over¬ 
positive  and  dogmatic  habit  of  mind,  or  feel  perhaps  that  a  condition  of  doubt  will  be 
regarded  as  evidence  of  mental  inferiority.  It  is  at  the  point  where  examination  and  test 
enter  into  investigation  that  the  difference  between  reflective  thought  and  bad  thinking 
comes  in.  To  be  genuinely  thoughtful,  we  must  be  willing  to  sustain  and  protract  that 
state,  of  doubt  which  is  the  stimulus  to  thorough  inquiry,  so  as  not  to  accept  an  idea  or 

30 

make  positive  assertion  of  a  belief  until  justifying  reasons  have  been  found. 

Many  definitions  and  descriptions  of  critical  thinking  use  “reflective”  or  “reflective  thinking”  as 
a  component  in  the  definition.  Some  scholars  also  quote  from  Dewey’s  qualitative  description  of 
being  sufficiently  critical  when  thinking  in  their  definitions  of  critical  thinking. 

Henry  Hazlitt  used  a  variety  of  modifiers  for  the  word  “thinking”  in  his  1916  book  entitled 
Thinking  as  a  Science,  including  good  thinking,  methodic  thinking,  positive  and  constructive 

3 1 

thinking,  loose  thinking,  thinking  with  a  purpose,  uncritical  thinking,  and  critical  thinking. 
Although  critical  thinking  was  included  in  the  book,  the  term  does  not  appear  to  have  had  any 
special  significance  in  1916. 

Teaching  to  Think  by  Julius  Boraas,  published  in  1924,  was  the  first  book  I  found  that  had  an 
explicit  focus  on  the  topic  of  critical  thinking.  The  chapter  called  “The  Development  of  Skill  in 
Critical  Thinking”  explained  the  meaning  of  criticism  and  applied  the  concept  of  criticism  to 
thinking. 

But  why  is  it  that  the  critical  thinking  of  pupils,  and  of  a  large  number  of  adults  too,  is  so 
crude?  According  to  the  dictionary  criticism  is  the  “act  or  art  of  judging  by  some 

37 

standard.”  Note  that  the  dictionary  calls  it  an  “art.” 

Boraas  also  provided  a  more  detailed  explanation  of  critical  thinking: 

When  all  things  have  been  considered,  critical  thinking  is  found  to  make  three  important 
demands:  it  insists  on  clearness.  The  critical  thinker  is  always  asking  such  questions  as: 
Just  what  did  we  see?  Just  what  were  the  sounds  we  heard?  What  were  the  exact  words 
used?  What  did  he  really  mean  to  say?  What  did  he  do?  What  did  he  mean  to  do? 
Critical  thinking  makes  a  persistent  demand  for  all  the  actual  facts  of  the  case.  It  has  no 
patience  with  merely  alleged  facts.  It  rebels  against  all  reports  which  are  “made  to  order” 
and  “fixed”  to  suit  the  supposed  needs  of  the  hearer.  Finally,  critical  thinking  is  effective 
thinking,  it  demands  results.  It  is  opposed  to  speculations  which  lead  nowhere.  It  objects 
to  everything  that  is  irrelevant  and  immaterial.  To  some  people  critical  thinking  is 
merely  the  type  of  thinking  that  demands  clearness  and  truth.  It  means  more  than  that.  It 


30  Dewey,  p.  16. 

31Hazlitt,  H.  Thinking  as  a  Science,  E.  P.  Dutton  &  Co.,  New  York,  NY,  1916. 

’2  Boraas,  J.  Teaching  to  Think,  Macmillan,  New  York,  NY,  1922,  p.  202. 
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insists  that  the  particular  truth  which  we  hold  be  one  that  has  value  in  reaching  a 
conclusion.33 

The  first  critical  thinking  assessment,  now  called  the  Watson-Glaser  Critical  Thinking 
Assessment  (WGCTA),  became  available  in  an  experimental  form  in  1938. 34 

The  US  Educational  Policies  Commission  highlighted  the  importance  of  critical  thinking,  which 
they  called  critical  judgment,  in  1938. 

In  other  words,  critical  judgment  is  developed  just  as  is  the  ability  to  play  chess,  or  to 
read  a  book,  or  to  solve  problems  in  geometry;  that  is,  by  long  and  continuous  practice 
under  the  criticism  of  someone  qualified  to  evaluate  the  decisions.  The  child  must  leam 
the  value  of  evidence.  He  must  acquire  a  reverence  for  facts,  must  desire  to  find  them, 
and  must  learn  where  they  can  most  likely  be  secured.  There  are  certain  sources  of  facts, 
certain  repositories  of  knowledge,  that  have  been  authenticated  through  the  years.  The 
student  must  learn  what  they  are  and  acquire  the  technique  of  using  them,  and  develop 
the  habit  of  turning  to  them  when  called  upon  to  solve  problems.  He  must  learn  to  defer 
judgment,  to  consider  motives,  to  appraise  evidence,  to  classify  it,  to  array  it  on  one  side 
or  the  other  of  his  question,  and  to  use  it  in  drawing  conclusions.  This  is  not  the  result  of 
a  special  course  of  study,  or  of  a  particular  part  of  the  educative  procedure;  it  results  from 

35 

every  phase  of  learning  and  characterizes  every  step  of  thinking. 

By  1939  critical  thinking  as  an  educational  ideal  was  sufficiently  understood  and  considered 
important  enough  that  the  New  York  Times  reported  that  Cornell  University  applied  for  and 
received  a  grant  to  study  critical  thinking  in  the  classroom: 

This  Fall  Cornell  University,  aided  by  an  $18,000  grant  from  the  General  Education 
Board,  is  starting  a  three-year  experiment  to  discover  whether  a  capacity  for  critical 
thinking  about  social  problems  can  be  developed  in  high  school  pupils.36 

At  the  beginning  of  the  20th  Century,  critical  thinking  was  just  one  of  many  terms  used  for  good 
thinking.  By  the  end  of  the  century  it  was  the  only  tenn  related  to  thinking  that  was  listed  as  part 
of  the  National  Educational  Goals: 

The  proportion  of  college  graduates  who  demonstrate  an  advanced  ability  to  think 

37 

critically,  communicate  effectively,  and  solve  problems  will  increase  substantially. 

The  growth  of  critical  thinking  as  the  primary  term  referring  to  good  thinking  can  also  be 
demonstrated  by  using  the  Google  search  capability  known  as  the  Ngram  Viewer.  The  Ngram 
Viewer  searches  for  specific  terms  in  a  corpus  of  books  for  a  specified  period  of  time  to 
determine  the  usage  of  the  terms.38  In  order  to  create  the  Ngram  view  shown  in  Figure  1, 1  used 


33  Boraas,  p.  215. 

34  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  p. 

87. 

35  Educational  Policies  Commission.  The  Purposes  of  Education  in  American  Democracy,  National  Educational 
Association  of  the  United  States,  Washington,  D.C.,  1938,  p.  35. 

36  Anonymous.  "Cornell  Plans  Test  of  Pupil-Thinking,"  New  York  Times,  1939,  September  17,  pp.  57. 

’7  “Panel  Goal  6:  Adult  Literacy  and  Lifelong  Learning,”  National  Education  Goals  Panel.  Available: 
http://govinfo.librarv.unt.edu/negp/page3-13.htm.  Viewed:  March  17,  2011. 

38  Google  Books  Ngram  Viewer,  Google  Inc.  Available: 

http://books.  google. com/ngrams/graph?content=straight+thinking%2Creflective+thinking%2Ccritical+thinking%2C 

clear+thinking%2Csound+thinking&year  start- 1 900&year  end=20 1 0&corpus=0&smoothing=3  Viewed:  February 
8,  2012.  See  also: 
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the  terms  clear  thinking,  straight  thinking,  critical  thinking,  sound  thinking,  and  reflective 
thinking. 


|  straight  thinking  |  reflective  thinking  |  critical  thinking  |  clear  thinking  |  sound  thinking 


Figure  1:  Usage  of  Terms  Related  to  Good  Thinking 

As  the  Ngram  view  shows,  in  the  1920s:  critical  thinking  was  the  least  used  term;  it  became  the 
dominant  term  by  the  1960s  and  has  been  used  even  more  frequently  from  about  1980  to  the 
present.  It  is  not  clear  from  the  literature  I  reviewed  what  caused  critical  thinking  to  become  the 
dominant  term  used  to  describe  “good  thinking.” 

4.1.2.  Defining  Critical  Thinking 

I  started  my  research  by  trying  to  find  a  definition  of  critical  thinking  in  order  to  understand 
exactly  what  critical  thinking  is.  However,  a  single,  universally  accepted  definition  of  critical 
thinking  does  not  exist.  Sufficient  differences  existed  in  the  meaning  of  the  term  as  early  as  1987 
that  Barry  Beyer  commented: 

The  term  critical  thinking  is  one  of  the  most  abused  terms  in  our  thinking  skills 
vocabulary.  Generally  it  means  whatever  its  users  stipulate  it  to  mean.  In  some  circles 
the  term  critical  thinking  is  used  to  mean  all  thinking  operations,  from  decision  making  to 
analysis  of  part-whole  relationships  to  interpreting.  In  other  circles  it  means  the  skills 
drawn  from  Bloom's  taxonomy.  Yet  critical  thinking  is  not  to  be  considered  as 
encompassing  all,  or  identical  to  any,  of  these  operations.  Critical  thinking,  for  example, 
is  obviously  not  the  same  as  recall.  Neither  is  critical  thinking  synonymous  with  decision 

39 

making  or  problem  solving. 

The  recognition  that  a  universally  accepted  definition  of  critical  thinking  does  not  exist  continues 
to  the  present  time.  Condon  and  Kelly-Riley  described  the  problem  this  way: 

The  current  literature  on  critical  thinking  is  rife  with  conflict  and  competing  ideologies.40 


Jean-Baptiste  Michel*,  Yuan  Kui  Shen,  Aviva  Presser  Aiden,  Adrian  Veres,  Matthew  K.  Gray,  William  Brockman, 
The  Google  Books  Team,  Joseph  P.  Pickett,  Dale  Hoiberg,  Dan  Clancy,  Peter  Norvig,  Jon  Orwant,  Steven  Pinker, 
Martin  A.  Nowak,  and  Erez  Lieberman  Aiden*.  Quantitative  Analysis  of  Culture  Using  Millions  of  Digitized  Books. 
Science  (Published  online  ahead  of  print:  12/16/2010) 

39  Beyer,  B.K.  Practical  Strategies  for  the  Teaching  of  Thinking,  Allyn  and  Bacon,  Boston,  MA,  1987,  p.  32. 
40Condon,  W.  &  Kelly-Riley,  D.  "Assessing  and  Teaching  What  We  Value:  The  Relationship  between  College- 
Level  Writing  and  Critical  Thinking  Abilities,"  Assessing  Writing,  Vol.  9,  No.  1,  2004,  p.  64. 
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My  research  confirmed  the  conflict  and  competing  ideologies.  I  retrieved  106  documents  that 
contained  at  least  one  definition  of  critical  thinking,  and  many  contained  multiple  definitions.  I 
found  219  definitions,  including  many  duplicates,  in  the  documents  I  reviewed.  I  identified  118 
unique  definitions.41 

The  high  number  of  duplicate  definitions  was  primarily  due  to  the  following: 

•  Many  documents  began  with  a  list  of  definitions  the  author  found.  The  author  then 
discussed  which  definition  was  preferred  or  why  all  definitions  were  inadequate  and 
a  new  definition  was  needed. 

•  The  originator  of  a  definition  wrote  multiple  books  or  articles,  each  referencing  his 
or  her  preferred  definition. 

•  Some  authors  adopted  one  of  the  extant  definitions  and  used  this  definition  in  one 
or  more  documents. 

Several  factors  contributed  to  the  high  number  of  unique  definitions: 

•  Although  a  few  scholars  have  referred  to  and  used  definitions  provided  by  others, 
most  critical  thinking  scholars  developed  their  own  unique  definitions  as  part  of 
their  scholarly  work.  Some,  such  as  Richard  Paul  with  the  Foundation  for  Critical 
Thinking,  developed  multiple  definitions,  further  confusing  the  issue. 

•  Research  and  scholarship  on  critical  thinking  has  been  conducted  in  multiple 
academic  fields.  Each  of  these  fields  has  its  own  approach  to  the  topic,  its  own  lens 
through  which  to  view  critical  thinking,  and  its  own  intended  application  of 
thinking  critically.  A  definition,  such  as  Sternberg’s,  from  the  cognitive 
psychology  tradition  is  different  than  a  definition  from  the  educational  tradition, 
such  as  Ennis’. 

•  Many  entities  with  an  interest  in  applying  critical  thinking  developed  their  own 
definition  after  reviewing  the  multitude  of  definitions  available.  Examples  include 
Baker  University,42  South  Carolina,43  the  nursing  profession,44  and  the  intelligence 
community.45 

An  example  of  the  challenge  that  exists  in  reaching  agreement  on  a  definition  comes  from  the 
1990  American  Philosophical  Association  (APA)  study  of  critical  thinking.46  One  outcome  of 
this  study  was  a  definition  of  critical  thinking.  Yet  even  though  47  scholars  from  multiple  fields 
agreed  to  what  has  been  called  the  “APA  consensus  definition  of  critical  thinking,”  the  only 


41  See  Appendix  A  for  a  complete  list  of  the  definitions  I  retrieved  during  my  research. 

42  Hatcher,  D.L.  Critical  Thinking  and  the  Liberal  Arts,  Baker  University,  Baldwin  City,  KS,  1990;  Hatcher,  D.L. 
Critical  Thinking:  A  New  Definition  and  Defense,  Baker  University,  Baldwin  City,  KS,  2000. 

43  Cook,  P.,  Johnson,  R.,  Moore,  P.,  Myers,  P.,  Pauly,  S.,  Pendarvis,  F.,  Prns,  J.  &  Ulmer-Sottong,  L.  Critical 
Thinking  Assessment:  Measuring  a  Moving  Target.  Report  &  Recommendations  of  the  South  Carolina  Higher 
Education  Assessment  Network  Critical  Thinking  Task  Force,  South  Carolina  Higher  Education  Assessment 
Network,  Rock  Hill,  SC,  1996. 

44  Staib,  S.  "Teaching  and  Measuring  Critical  Thinking,"  Journal  of  Nursing  Education,  Vol.  42,  No.  11,  2003,  pp. 
498-508. 

45  For  example  Hendrickson,  N.  "Critical  Thinking  in  Intelligence  Analysis,"  International  Journal  of  Intelligence 
and  Counterintelligence,  Vol.  21,  No.  4,  2008,  pp.  679-693. 

46  Facione,  P.A.  Critical  Thinking:  A  Statement  of  Expert  Consensus  for  Purposes  of  Educational  Assessment  and 
Instruction.  Research  Findings  and  Recommendations.  American  Philosophical  Association,  Newark,  DE,  1990. 
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participant  who  continues  to  use  the  consensus  definition  is  the  facilitator  of  the  process,  Peter 
Facione. 


The  definitions  I  identified  varied  widely  in  their  understandability  and  their  practicality.  Some 
were  simplistic;  others  were  overly  complex.  In  the  search  for  practical  definitions  or 
descriptions,  that  is,  those  that  enable  the  reader  or  practitioner  to  understand  what  distinguishes 
critical  thinking  from  non-critical  thinking,  I  found  two  approaches.  Critical  thinking  has  been 
perceived  as  a  noun  -  a  thing  to  be  defined  or  described.  A  more  informative  explanation  came 
from  those  who  also  described  the  activity  of  thinking  critically  -  the  act  of  thinking  with  the 
qualitative  descriptor  critical  applied  to  it  as  Dewey  described. 

Despite  the  differences  in  definitions,  commonalities  do  exist  between  definitions.  The  most 
common  themes  that  appeared  in  the  definitions  and  descriptions  include: 

•  Deliberate/intentional/active  approach  to  thinking 

•  Systematic  in  thinking;  focused;  disciplined,  skilled 

•  Persistent,  not  giving  up  too  soon 

•  Focus  on  objectivity,  remaining  open  to  what  the  evidence  tells  you 

•  Interpretation  of  infonnation 

•  Evaluation  of  all  infonnation 

•  Synthesis  of  valid  information 

•  Analysis  of  information 

•  Inference,  judgment,  reaching  a  conclusion  that  can  be  justified 
Some  definitions  also  include  the  following: 

•  Explanation  of  results  and  evidence  used  to  reach  results 

•  Assessment  of  assumptions 

•  Formal  logic 

•  Skepticism 

•  Creativity 

•  Reflection 

•  Use  of  intellectual  standards  or  criteria 

•  Metacognition  -  thinking  about  one’s  thinking 

I  chose  a  sample  of  definitions  for  inclusion  in  this  report  to  illustrate  the  diversity  of 
perspectives  in  defining  critical  thinking.  The  samples  reflect  two  categories: 

1 .  Scholarly  definitions  written  by  those  who  are  primarily  focused  on  academic  research 
and  theories 

2.  Applied  definitions  written  by  those  who  are  attempting  to  actually  think  critically, 
including  those  in  the  intelligence  analysis  literature 

Scholarly  Definitions :  Edward  Glaser  was  the  source  of  the  earliest  formal  definition  of  critical 
thinking  that  I  was  able  to  locate.  In  1941,  Glaser,  a  professor  of  education  at  Columbia 
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University  and  co-author  of  the  Watson-Glaser  Critical  Thinking  Test/7  wrote  about  his  research 
in  teaching  students  to  think  critically.  His  definition  of  critical  thinking  is: 

The  ability  to  think  critically,  as  conceived  in  this  volume,  involves  three  things:  (1)  an 
attitude  of  being  disposed  to  consider  in  a  thoughtful  way  the  problems  and  subjects  that 
come  within  the  range  of  one's  experiences,  (2)  knowledge  of  the  methods  of  logical 

4.8 

inquiry  and  reasoning,  and  (3)  some  skill  in  applying  those  methods. 

Glaser’s  definition  includes  three  themes  that  recur  in  the  literature: 

•  The  importance  of  the  disposition  to  think  critically  because  skills  are  insufficient 
without  a  critical  attitude. 

•  The  importance  of  skills  in  successfully  thinking  critically. 

•  The  role  of  logic  in  critical  thinking. 

Glaser  continued  immediately  with  an  explanation  of  how  to  think  critically.  He  quoted  directly 
from  Dewey’s  definition  of  reflective  thinking,  thus  linking  the  critical  thinking  tradition 
formally  with  Dewey’s  reflective  thinking.  Glaser’s  description  clarifies  the  act  of  thinking 
critically  far  beyond  Dewey’s  definition: 

Critical  thinking  calls  for  a  persistent  effort  to  examine  any  belief  or  supposed  form  of 
knowledge  in  the  light  of  the  evidence  that  supports  it  and  the  further  conclusions  to 
which  it  tends.  It  also  generally  requires  ability  to  recognize  problems,  to  find  workable 
means  for  meeting  those  problems,  to  gather  and  marshal  pertinent  information,  to 
recognize  unstated  assumptions  and  values,  to  comprehend  and  use  language  with 
accuracy,  clarity,  and  discrimination,  to  interpret  data,  to  appraise  evidence  and  evaluate 
arguments,  to  recognize  the  existence  (or  non-existence)  of  logical  relationships  between 
propositions,  to  draw  warranted  conclusions  and  generalizations,  to  put  to  test  the 
conclusions  and  generalizations  at  which  one  arrives,  to  reconstruct  one’s  patterns  of 
beliefs  on  the  basis  of  wider  experience,  and  to  render  accurate  judgments  about  specific 
things  and  qualities  in  everyday  life.49 

Glaser’s  definition  was  directly  referenced  in  three  other  documents. 

Although  many  scholars  noted  the  importance  of  Glaser’s  work,  authors  did  not  quote  his 
definition  frequently.  Instead,  the  most  commonly  cited  scholarly  definitions  of  critical  thinking 
came  from  three  sources: 

•  The  work  of  philosopher  and  educator  Robert  F  inn  is 

•  The  work  of  the  APA  expert  panel  which  resulted  in  the  APA  consensus  definition 
of  critical  thinking 

•  The  Foundation  for  Critical  Thinking 


47  Watson,  G.  &  Glaser,  E.M.  Watson-Glaser  II  Critical  Thinking  Appraisal  Technical  Manual  and  User's  Guide, 
NCS  Pearson,  Inc.,  San  Antonio,  TX,  2009. 

48  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  p.  5. 

49  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  pp. 
5-6. 
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Robert  Ennis,  author  of  the  Cornell  Critical  Thinking  Test50  and  the  Ennis  Weir  Critical 
Thinking  Essay  Test,51  has  written  extensively  about  critical  thinking.  He  developed  two 
definitions,  the  first  of  which  was  proposed  in  1962. 

As  a  root  notion  critical  thinking  is  taken  to  be  the  correct  assessing  of  statements. 

Ennis  explained  the  evolution  of  his  thinking  in  1993: 

A  definition  of  critical  thinking  that  I  at  one  time  endorsed  is  that  critical  thinking  is  the 
correct  assessing  of  statements  (Ennis,  1962).  If  I  had  not  elaborated  this  definition,  it 
would  be  as  vague  as  Bloom's  taxonomy.  But  even  when  elaborated,  it  suffers  from 
excluding  creative  aspects  of  critical  thinking,  such  as  conceiving  of  alternatives, 
formulating  hypotheses  and  definitions,  and  developing  plans  for  experiments.  I  now 
think  the  contemporary  conception  of  critical  thinking  includes  these  things,  so  the 
"correct  assessing"  definition  is  more  narrow  than  standard  usage,  and  thus  could 
interfere  with  communication  among  proponents  of  critical  thinking. 

The  following  definition  seems  to  be  more  in  accord  with  contemporary  usage  and  thus,  I 
hope,  will  minimize  confusion  in  communication:  "Critical  thinking  is  reasonable 
reflective  thinking  focused  on  deciding  what  to  believe  or  do." 

Ennis  took  advantage  of  Dewey’s  concept  of  reflective  thinking  for  his  current  definition, 
continuing  the  connection  with  Dewey.  The  original  Ennis  definition  was  included  in  nine 
documents,  four  of  which  were  authored  by  Ennis.  Ennis’  current  definition  (sometimes  with 
slight  variations  such  as  “Critical  thinking  was  characterized  as  a  self-adjusting  process  of 
judging  what  to  believe  or  what  to  do  in  a  given  context.”)  was  included  in  23  documents,  six  of 
which  were  authored  by  Ennis. 

In  1990  Peter  Facione  led  an  effort  under  the  auspices  of  the  American  Philosophical  Association 
that  made  a  systematic  inquiry  into  the  current  state  of  critical  thinking  and  thinking  assessment. 
Facione  solicited  opinions  from  major  researchers  in  the  realm  of  critical  thinking  in  universities 
across  the  US.  The  researchers  represented  a  broad  range  of  disciplines  including  philosophy, 
psychology,  education,  social  sciences,  computer  science,  and  physics.  Early  in  the  effort  the 
participants  realized  they  needed  to  focus  on  clarifying  what  is  meant  by  the  tenn  critical 
thinking. 

A  clear  and  accurate  conceptualization  of  CT  is  absolutely  essential  for  the  development 
of  valid  CT  assessment  tools  and  effective  CT  instructional  programs.  With  this  in  mind, 
and  recognizing  that  divergent  conceptualizations  of  CT  have  hindered  curricular  and 
assessment  efforts,  early  in  the  Delphi  process  the  panel  decided  its  most  worthwhile 
contribution  could  be  the  articulation  of  a  clear  and  correct  conceptualization  of  CT.  The 
expert  panelists  devoted  their  major  effort  toward  that  end.54 

The  expert  panelists  reached  a  consensus  on  the  conceptualization  of  critical  thinking: 

50  Ennis,  R.H.,  Millman,  J.  &  Tomko,  T.N.  Cornell  Critical  Thinking  Tests  Level  X &  Level  Z  Manual,  5th  ed.,  The 
Critical  Thinking  Co.,  Seaside,  CA,  2005. 

51  Ennis,  R.H.  &  Weir,  E.  The  Ennis-Weir  Critical  Thinking  Essay  Test,  Midwest  Publications,  Pacific  Grove  CA, 
1985. 

52  Ennis,  R.H.  "A  Concept  of  Critical  Thinking,"  Harvard  Educational  Review,  Vol.  32,  No.  1,  1962,  p.  83. 

53  Ennis,  R.H.  "Critical  Thinking  Assessment,"  Theory  into  Practice,  Vol.  32,  No.  3,  1993,  pp.  179-180. 

Facione,  P.A.  Critical  Thinking:  A  Statement  of  Expert  Consensus  for  Purposes  of  Educational  Assessment  and 

Instruction.  Research  Findings  and  Recommendations.  American  Philosophical  Association,  Newark,  DE,  1990,  pp. 
8-9. 
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We  understand  critical  thinking  to  be  purposeful,  self-regulatory  judgment  which  results 
in  interpretation,  analysis,  evaluation,  and  inference,  as  well  as  explanation  of  the 
evidential,  conceptual,  methodological,  criteriological,  or  contextual  considerations  upon 
which  that  judgment  is  based.  CT  is  essential  as  a  tool  of  inquiry.  As  such,  CT  is  a 
liberating  force  in  education  and  a  powerful  resource  in  one’s  personal  and  civic  life.55 

After  providing  a  definition,  the  panel  then  described  thinking  critically: 

While  not  synonymous  with  good  thinking,  CT  is  a  pervasive  and  self-rectifying  human 
phenomenon.  The  ideal  critical  thinker  is  habitually  inquisitive,  well-informed,  trustful 
of  reason,  open-minded,  flexible,  fair-minded  in  evaluation,  honest  in  facing  personal 
biases,  prudent  in  making  judgments,  willing  to  reconsider,  clear  about  issues,  orderly  in 
complex  matters,  diligent  in  seeking  relevant  information,  reasonable  in  the  selection  of 
criteria,  focused  in  inquiry,  and  persistent  in  seeking  results  which  are  as  precise  as  the 
subject  and  the  circumstances  of  inquiry  pennit.  Thus,  educating  good  critical  thinkers 
means  working  toward  this  ideal.  It  combines  developing  CT  skills  with  nurturing  those 
dispositions  which  consistently  yield  useful  insights  and  which  are  the  basis  of  a  rational 
and  democratic  society.56 

The  definition  and  description  that  resulted  from  this  effort  continued  Glaser’s  focus  on  thinking 
skills  and  dispositions  as  important  factors  in  critical  thinking,  but  were  not  otherwise  linked  to 
earlier  definitions. 

It  is  interesting  to  note  that  while  Facione  has  embraced  this  definition  and  used  it  as  the  basis  for 
the  California  Critical  Thinking  Skills  Tests57  (CCTST)  and  the  California  Critical  Thinking 
Dispositions  Inventory)  (CCTDI)  many  of  the  participants  in  the  APA  study  continue  to  use 
their  own  definitions  or  conceptualizations  of  critical  thinking  rather  than  using  the  APA 
consensus  definition.  For  example,  Robert  Ennis  continues  to  use  his  definition  of  “reasonable 
reflective  thinking  focused  on  deciding  what  to  believe  or  do.”  Another  APA  participant, 

Richard  Paul  is  the  author  or  supporter  of  the  many  definitions  proposed  by  the  Foundation  for 
Critical  Thinking,  which  I  discuss  in  the  next  set  of  definitions. 

The  APA  consensus  definition  was  referenced  in  10  documents,  four  of  which  were  authored  by 
Facione. 

The  organization  that  is  known  today  as  the  Foundation  for  Critical  Thinking  was  established  in 
1980  as  the  Center  for  Critical  Thinking.  The  Center  is  now  part  of  the  Critical  Thinking 
Community  consisting  of  the  Foundation  for  Critical  Thinking,  the  Center  for  Critical  Thinking, 
the  National  Council  for  Excellence  in  Critical  Thinking,  and  the  International  Center  for  the 
Assessment  of  Higher  Order  Thinking.  When  I  searched  for  critical  thinking  using  common 
search  engines,  the  Foundation  for  Critical  Thinking  is  one  of  the  first  sites  returned  and  is 
therefore  likely  to  be  one  of  the  first  organizations  to  be  consulted  about  critical  thinking. 


55  Facione,  p.  3. 

56  Facione,  p.3. 

57  “California  Critical  Thinking  Skills  Tests,”  Insight  Assessment.  Available:  http://www.insightassessment.com/. 
Viewed:  April  29,  2010. 

58  “California  Critical  Thinking  Disposition  Inventory:  Scale  Descriptions,”  Insight  Assessment.  Available: 
http://www.insightassessment.com/Scales%20CCTDI.html,  Viewed:  October  9,  2009. 
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The  Foundation  has  provided  annual  conferences  and  workshops  on  critical  thinking  since  1980, 
published  numerous  reports,  and  published  a  series  of  miniature  guides  to  various  aspects  of 
critical  thinking.  The  Foundation  cites  at  least  eight  different  definitions  of  critical  thinking  in  its 
publications  and  on  its  web  site,  several  of  which  have  been  widely  quoted.  The  primary  authors 
of  Foundation  definitions,  when  an  author  is  noted,  are  Richard  Paul,  Linda  Elder,  and  Michael 
Scriven. 

The  most  widely  quoted  and  paraphrased  of  the  Foundation’s  definitions  is  one  of  the  least 
useful  found  in  the  entire  critical  thinking  literature.  It  owes  nothing  to  any  previous  tradition  in 
critical  or  reflective  thinking  definitions: 

Critical  thinking  is  the  art  of  analyzing  and  evaluating  thinking  with  a  view  to  improving 
it.59 

This  definition,  or  a  variation  of  it  such  as  “Thinking  about  our  thinking  while  we’re  thinking  in 
order  to  make  our  thinking  better,”  was  included  in  12  documents.  Three  of  these  documents 
were  authored  by  a  member  of  the  Foundation. 

The  second  most  widely  quoted  definition  from  the  Foundation  is  more  practical  and  is  linked  to 
both  Ennis’  current  definition  and  to  the  APA  consensus  definition. 

Critical  thinking  is  the  intellectually  disciplined  process  of  actively  and  skillfully 
conceptualizing,  applying,  analyzing,  synthesizing,  and/or  evaluating  information 
gathered  from,  or  generated  by,  observation,  experience,  reflection,  reasoning,  or 
communication,  as  a  guide  to  belief  and  action.60 

This  definition  was  included  in  ten  documents,  two  of  which  were  authored  by  a  member  of  the 
Foundation. 

Although  the  definitions  just  described  are  the  most  commonly  quoted  in  the  literature,  an 
examination  of  the  definitions  of  critical  thinking  from  different  academic  disciplines  is  useful 
for  illustrating  how  diverse  scholarly  perspectives  are. 

Robert  Sternberg  provided  a  psychologically-based  definition  in  his  1986  report,  Critical 
Thinking:  Its  Nature,  Measurement,  and  Improvement.  Sternberg’s  definition  can  be  described  as 
a  process-oriented  definition.  It  is  not  linked  to  any  of  the  earlier  definitions  and  provides  no  hint 
of  how  to  think  critically. 

Construed  broadly,  critical  thinking  comprises  the  mental  processes,  strategies,  and 
representations  people  use  to  solve  problems,  make  decisions,  and  learn  new  concepts.61 

Sternberg’s  definition  was  quoted  in  one  other  document. 

Diane  Halpern  provided  another  psychologically-based  definition: 

Critical  thinking  is  the  use  of  those  cognitive  skills  or  strategies  that  increase  the 
probability  of  a  desirable  outcome.  It  is  used  to  describe  thinking  that  is  purposeful, 
reasoned,  and  goal  directed  -  the  kind  of  thinking  involved  in  solving  problems, 


59  Paul,  R.  &  Elder,  L.  The  Miniature  Guide  to  Critical  Thinking  Concepts  and  Tools,  Foundation  for  Critical 
Thinking,  Dillon  Beach,  CA,  2007,  p.  4. 

60  Scriven,  M.  &  Paul,  R.  “Defining  Critical  Thinking,”  Foundation  for  Critical  Thinking.  Available: 
http://www.criticalthinking.org/page.cfm?PageID=410&CategoryID=51,  Viewed:  March  16,  2010. 

61  Sternberg,  R.J.  Critical  Thinking:  Its  Nature,  Measurement,  and  Improvement,  Yale  University,  New  Flaven,  CT, 
1986,  p.  1. 
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formulating  inferences,  calculating  likelihoods,  and  making  decisions,  when  the  thinker  is 
using  skills  that  are  thoughtful  and  effective  for  the  particular  context  and  type  of 
thinking  task.62 

Halpern’s  definition  was  quoted  in  six  other  documents. 

Educator  Joanne  Kurhss  wrote  a  report  on  critical  thinking  for  the  Association  for  the  Study  of 
Higher  Education.  She  provided  a  practical  definition  of  critical  thinking  and  followed  it 
immediately  with  a  description  of  thinking  critically. 

Critical  thinking  is  a  rational  response  to  questions  that  cannot  be  answered  definitively 
and  for  which  ah  the  relevant  information  may  not  be  available.  It  is  defined  here  as  an 
investigation  whose  purpose  is  to  explore  a  situation,  phenomenon,  question,  or  problem 
to  arrive  at  a  hypothesis  or  conclusion  about  it  that  integrates  ah  available  information 
and  that  can  therefore  be  convincingly  justified. 

In  critical  thinking,  ah  assumptions  are  open  to  question,  divergent  views  are 
aggressively  sought,  and  the  inquiry  is  not  biased  in  favor  of  a  particular  outcome. 

The  outcomes  of  a  critical  inquiry  are  twofold:  a  conclusion  (or  hypothesis)  and  the 
justification  offered  in  support  of  it.  These  outcomes  are  usually  set  forth  in  the  form  of 
an  argument,  defined  as  "the  sequence  of  interlinked  claims  and  reasons  that,  between 
them,  establish  the  content  and  force  of  the  position  for  which  a  particular  speaker  is 
arguing"  (Toulmin,  Rieke,  and  Janik  1979,  p.  13).  The  need  for  justification  arises  from 
the  ill-defined  nature  of  problems  to  which  the  term  "critical  thinking"  generally  applies. 
Because  conclusions  cannot  be  tested  (as  they  can  be  in  problem  solving),  the  arguer 
must  demonstrate  their  plausibility  by  offering  supporting  reasons  (Voss,  Tyler,  and 
Yengo  1983). 63 

Kurfiss’  definition  was  quoted  in  one  other  document. 

Philosophers  Alex  Fisher  and  Michael  Scriven  provided  a  definition  in  their  1997  critical 
thinking  textbook.  This  definition  was  followed  by  over  20  pages  of  text  explaining  each  term  in 
the  definition. 

Critical  thinking  is  skilled,  active  interpretation  and  evaluation  of  observations, 
communications,  information,  and  argumentation.64 

This  definition  was  quoted  in  a  later  book  written  by  Fisher. 

Brooke  Moore  and  Richard  Parker  also  included  a  philosophy-based  definition  in  their  critical 
thinking  textbook.  Note  that  their  definition  specifically  excludes  inference,  an  important 
component  of  critical  thinking  that  appears  in  other  descriptions  and  is  an  essential  aspect  of 
intelligence  analysis. 


62  Halpern,  D.F.  Thought  and  Knowledge:  An  Introduction  to  Critical  Thinking,  Lawrence  Erlbaum  Associates, 
Mahwah,  NJ,  2003,  p.  6. 

63  Kurfiss,  J.G.  Critical  Thinking:  Theory,  Research,  Practice,  and  Possibilities,  ASHE-ERIC  Higher  Education 
Report  No.  2,  Association  for  the  Study  of  Higher  Education,  Washington,  D.C.,  1988,  p.  2. 

64  Fisher,  A.  &  Scriven,  M.  Critical  Thinking:  Its  Definition  and  Assessment,  Edgepress,  Point  Reyes,  CA,  1997,  p. 
20. 
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But,  boiled  down,  critical  thinking  is  the  careful  application  of  reason  in  the 
determination  of  whether  a  claim  is  true.  Notice  that  it  isn't  so  much  coming  up  with 
claims,  true  or  otherwise,  that  constitutes  critical  thinking;  it's  the  evaluation  of  claims, 
however  we  come  up  with  them.65 

THE  BASICS:  CLAIMS,  ISSUES,  AND  ARGUMENTS 

In  the  next  few  pages,  we'll  introduce  the  basic  building  blocks  of  critical  thinking: 
claims,  issues,  and  arguments.  Identifying  these  elements,  including  separating  them  out 
from  embellishments  and  impostors,  and  analyzing  and  evaluating  them  are  what  critical 
thinking  is  all  about.66 

No  other  documents  referenced  this  specific  definition. 

It  should  be  noted  that  regardless  of  discipline,  each  author  of  a  scholarly  definition,  both  those 
discussed  here  and  those  included  in  Appendix  A,  stated  that  his  or  her  definition  is  the  correct 
definition. 

Applied  Definitions:  Individuals  who  intend  to  apply  critical  thinking  in  their  work  are  faced 
with  a  dilemma:  how  to  determine  which,  of  the  many  definitions  of  critical  thinking  that  exist,  is 
the  correct  definition?  Which  definition  should  be  used  as  the  basis  for  the  application  of  critical 
thinking  in  their  work?  Depending  on  how  extensive  their  research  was  before  making  a 
decision,  they  may  have  found  anywhere  from  a  few  definitions  to  over  one  hundred  as  we  found 
in  this  search.  Rather  than  choosing  an  existing  definition,  many  of  the  organizations 
considering  the  application  of  critical  thinking  opted  to  just  develop  their  own  definition. 

The  next  three  examples  illustrate  the  varying  approaches  taken  when  an  organization  opts  to 
develop  its  own  critical  thinking  definition.  An  organization  might: 

•  combine  multiple  definitions  into  one  new  definition 

•  reject  the  existing  definitions  to  develop  an  entirely  new  definition 

•  conduct  a  Delphi  study  similar  to  the  APA  study  discussed  earlier  in  order  to 
develop  a  new  definition 

In  addition  to  providing  examples  of  the  above  approaches,  this  section  of  the  report  also 
presents  a  sample  of  the  applied  definitions  developed  for  intelligence  analysis. 

Patricia  Cook  et  al.  provided  a  combined  definition  adapted  from  those  of  Beyer,  Facione, 
McPeck,  Sternberg,  and  Ennis  in  a  report  for  the  South  Carolina  Higher  Education  Assessment 
network.  This  definition  consists  of  terms  used  by  the  referenced  scholars  assembled  into  one 
statement. 

Critical  thinking  is  a  reflective,  systematic,  rational,  and  skeptical  use  of  cognitive 
representations,  processes,  and  strategies  about  beliefs,  problems,  and/or  courses  of 
action.67 


65  Moore,  B.N.  &  Parker,  R.  Critical  Thinking,  9th  ed.,  McGraw-Hill,  Boston,  MA,  2009,  p.  3. 

66  Moore  &  Parker,  p.  5. 

67  Cook,  P.,  Johnson,  R.,  Moore,  P.,  Myers,  P.,  Pauly,  S.,  Pendarvis,  F.,  Pius,  J.  &  Ulmer-Sottong,  L.  Critical 
Thinking  Assessment:  Measuring  a  Moving  Target.  Report  &  Recommendations  of  the  South  Carolina  Higher 
Education  Assessment  Network  Critical  Thinking  Task  Force,  South  Carolina  Higher  Education  Assessment 
Network,  Rock  Hill,  SC,  1996,  p.  6. 
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Donald  Hatcher,  leader  of  a  critical  thinking  project  at  Baker  University,  developed  a  new 
definition  after  deeming  the  existing  scholarly  definitions  inadequate.  His  recommended 
definition  is  very  similar  to  Kurfiss’  definition  and  is  related  to  Dewey’s  description  of  critical 
reflection.  It  is  focused  on  outcome  rather  than  skill  or  process: 

Critical  thinking  is  thinking  that  attempts  to  arrive  at  a  judgment  only  after  honestly 
evaluating  alternatives  with  respect  to  available  evidence  and  arguments.68 

In  2000,  members  of  the  nursing  profession  used  a  Delphi  study  to  develop  a  nursing-specific 
definition  of  critical  thinking.  This  definition  is  more  focused  on  dispositions  and  skills  than  on 
outcomes. 

Critical  thinking  is  an  essential  component  of  professional  accountability  and  quality 
nursing  care.  Critical  thinkers  in  nursing  exhibit  these  habits  of  the  mind:  confidence, 
contextual  perspective,  creativity,  flexibility,  inquisitiveness,  intellectual  integrity, 
intuition,  open-mindedness,  perseverance,  and  reflection.  Critical  thinkers  in  nursing 
practice  the  cognitive  skills  of  analyzing,  applying  standards,  discriminating,  information 
seeking,  logical  reasoning,  predicting  and  transforming  knowledge.69 

Within  the  intelligence  analysis  literature,  2 1  documents  contained  definitions  for  critical 
thinking  in  intelligence  analysis,  with  a  total  of  27  definitions  and  13  uniquely  new  definitions. 
Several  of  the  documents  included  multiple  definitions,  and  seven  of  the  2 1  documents 
referencing  scholarly  definitions  from  the  APA,  Halpem,  and  the  Foundation  for  Critical 
Thinking. 

A  sample  of  the  intelligence  analysis  definitions  follows.  The  first  two  definitions  share 
common  themes  with  each  other  and  with  the  scholarly  definitions.  The  third  definition 
represents  a  new  understanding  of  critical  thinking. 

Edward  Waltz  provided  a  definition  in  Knowledge  Management  in  the  Intelligence  Enterprise : 

Critical  thinking  is  based  on  the  application  of  a  systematic  method  to  guide  the 
collection  of  evidence,  reason  from  evidence  to  argument,  and  apply  objective  decision¬ 
making  judgment  (Table  4.10).  The  systematic  methodology  assures  completeness 
(breadth  of  consideration),  objectivity  (freedom  from  bias  in  sources,  evidence, 
reasoning,  or  judgment),  consistency  (repeatability  over  a  wide  range  of  problems),  and 
rationality  (consistency  with  logic).  In  addition,  critical  thinking  methodology  requires 
the  explicit  articulation  of  the  reasoning  process  to  allow  review  and  critique  by  others.70 

The  training  course,  Analysis  101,  referenced  several  definitions  before  providing  its  own: 

Critical  thinking  is  quality  control  for  problem  solving  and  reasoning.  Critical  thinking 
enables  you  to  judge  the  accuracy  and  relevance  of  your  information,  categorize 
information  as  assumption,  fact,  or  inference,  identify  gaps  in  your  knowledge,  recognize 
factors  that  influence  your  objectivity  and  the  objectivity  of  others,  and  judge  the  validity 
of  your  arguments.71 


6S  Hatcher,  D.L.  Critical  Thinking:  A  New  Definition  and  Defense,  Baker  University,  Baldwin  City,  KS,  2000,  p.  3. 

69  Staib,  S.  "Teaching  and  Measuring  Critical  Thinking,"  Journal  of  Nursing  Education,  Vol.  42,  No.  11,  2003,  p. 
499. 

70  Waltz,  E.  Knowledge  Management  in  the  Intelligence  Enterprise,  Artech  House,  Boston,  MA,  2003,  p.  144. 

71  Anonymous.  Analysis  101.  Participant  Guide  2008,  p.  1 1. 
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Intelligence  Community  Directive  (ICD)  Number  610  contains  the  competency  directories  for  the 
IC  workforce.  One  competency  defined  for  individual  contributors  and  for  supervisors  is  critical 
thinking  .  The  critical  thinking  definition  and  explanation  in  Table  1  comes  from  the  list  of 
competencies  for  non-supervisory  IC  employees  at  GS-15  and  below. 


Table  1:  ICD  610  -  Critical  Thinking  Competencies  for  Non-Supervisory  Personnel 


Critical 

Thinking 

IC  employees  are  expected  to  use  logic,  analysis,  synthesis,  creativity, 
judgment,  and  systematic  approaches  to  gather,  evaluate,  and  use  multiple 
sources  of  information  to  effectively  inform  decisions  and  outcomes. 

Creative 

Thinking 

Develops  new  insights  into  situations  and  applies  innovative  solutions  to 
problems  and  to  improve  processes.  Designs  new  methods  and  tools  where 
established  methods  and  procedures  are  inapplicable,  unavailable,  or 
ineffective. 

Exploring 

Alternatives 

Seeks  out,  evaluates,  and  integrates  a  variety  of  perspectives.  Seeks  to 
increase  own  and  others’  understanding  of  an  issue  based  on  new 
information  and  alternative  perspectives.  Listens  to  and  shows  appreciation 
for  alternative  ideas  and  approaches. 

Enterprise 

Perspective 

Understands  the  interrelationships  among  organizations  and  components  of 
the  IC.  Understands  how  one’s  own  work  impacts,  and  is  impacted  by,  the 
mission  and  operations  of  IC  organizations  and  components,  and  uses  this 
information  to  maximize  contribution  to  mission  accomplishment. 

Situational 

Awareness 

Maintains  awareness  of  changing  conditions,  current  events,  and  cultural 
and  historical  contexts  as  they  affect  one’s  own  work. 

Synthesis 

Identifies  and  uses  principles,  rules,  and  relationships  to  construct 
arguments  or  interpret  facts,  data,  or  other  information.  Dissects  problems 
into  meaningful  parts  and  uses  logic  and  judgment  to  detennine  accuracy 
and  relevance  of  data.  Identifies  and  reconciles  gaps,  uncertainties,  and  key 
assumptions  of  data.  Integrates  evidence/information,  evaluates  and 
prioritizes  alternatives,  and  assesses  similarities  and  differences  in  data  to 
develop  findings  and  conclusions.  Understands  potential  implications  of 
these  findings  or  conclusions. 

Three  documents  referenced  the  definition  for  individual  contributors. 


The  critical  thinking  definition  and  description  in  Table  2  is  excerpted  from  the  core 

7T 

competencies  for  supervisory  and  managerial  IC  employees  at  GS-15  and  below. 


72  Office  of  the  Director  of  National  Intelligence.  Intelligence  Community  Directive  Number  61 0.  Competency 
Directories  for  the  Intelligence  Community  Workforce.  2008,  p.  B-2. 

7l  Office  of  the  Director  of  National  Intelligence,  pp.  C-l  -  C-2. 
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Table  2:  ICD  610  -Critical  Thinking  Competencies  for  Supervisory  and  Managerial 

Personnel 


Critical 

Thinking 

IC  employees  are  expected  to  use  logic,  analysis,  synthesis,  creativity, 
judgment,  and  systematic  approaches  to  gather,  evaluate,  and  use  multiple 
sources  of  information  to  effectively  inform  decisions  and  outcomes.  In 
addition,  IC  supervisors  are  expected  to  establish  a  work  environment 
where  employees  feel  free  to  engage  in  open,  candid  exchanges  of 
information  and  diverse  points  of  view. 

Decisiveness 

Identifies  and  understands  issues,  problems,  and  opportunities.  Compares 
data  from  different  sources  throughout  the  IC  and  external  agencies  to  draw 
conclusions.  Chooses  an  approach,  develops  solutions,  and  takes  action 
consistent  with  IC  values,  available  facts,  constraints,  and  probable 
consequences.  Makes  appropriate  decisions  in  a  timely  manner  even  when 
data  are  limited  or  solutions  produce  unpleasant  consequences. 

Flexibility 

Is  open  to  change  and  new  information;  rapidly  adapts  to  new  information, 
changing  conditions,  or  unexpected  obstacles. 

Problem 

Solving 

Identifies  and  analyzes  problems;  weighs  relevance  and  accuracy  of 
information;  generates  and  evaluates  alternative  solutions;  and  makes 
rec  ommendations . 

According  to  ICD  610,  senior  officers  do  not  have  a  critical  thinking  competency  expectation, 
although  systems  thinking  and  strategic  thinking  are  identified  as  required  sub-skills. 


The  many  definitions  reviewed  in  order  to  produce  this  report  illustrate  one  of  the  primary 
findings  of  this  study.  Both  inside  and  outside  the  intelligence  community,  a  common  definition 
of  critical  thinking  does  not  exist.  The  wide  variations  in  the  understandability  of  the  definitions 
and  the  range  of  cognitive  skills  and  activities  described  make  it  difficult  to  apply  the  concepts  in 
order  to  think  critically  .  For  example: 

•  Just  what  is  “reasonable  reflective  thinking”  as  defined  by  Ennis? 

•  How  can  someone  “think  about  his  thinking  while  he  is  thinking  in  order  to  make 
his  thinking  better”  as  described  by  the  Foundation  for  Critical  Thinking? 

•  What  are  the  “cognitive  skills  or  strategies  that  increase  the  probability  of  a 
desirable  outcome”  as  described  by  Kuhn? 

•  How  important  is  inference  in  critical  thinking? 

•  What  is  the  contribution  of  creativity  to  thinking  critically? 

4.1.3.  Critical  Thinking  and  Logic 

Another  factor  contributing  to  the  confusion  surrounding  critical  thinking  is  its  relationship  to 
philosophy  and  logic.  Critical  thinking  is  sometimes  perceived  as  being  the  same  as  formal 
logic.74  One  result  of  this  perception  is  that  many  universities  satisfy  critical  thinking 
requirements  through  philosophy  courses.  However,  logic  alone  does  not  enable  an  individual  to 
become  an  adept  critical  thinker.  For  example,  consider  that  both  of  the  philosophy-based 
definitions  discussed  earlier  in  this  report  exclude  the  concept  of  inference,  often  considered  an 
important  component  of  critical  thinking. 


74  For  example,  see  Glaser’s  definition  in  the  definitions  section  of  this  report  which  says  one  element  of  critical 
thinking  is  knowledge  of  logical  inquiry. 
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Max  Black  provided  an  outstanding  example  of  the  perception  that  critical  thinking  equals  logic 
in  his  book  Critical  Thinking.  The  book’s  subtitle  is  “An  Introduction  to  Logic  and  Scientific 
Method.”  Although  the  book  is  titled  Critical  Thinking,  Black  uses  the  tenn  critical  thinking 
only  four  times  in  the  text.  Rather,  Black  wrote  the  following  about  logic: 

LOGIC  can  be  briefly  defined  as  the  study  of  reasoning.  The  study  of  any  subject  calls 
for  thought,  and  every  student  is,  or  ought  to  be,  a  thinker;  but  he  is  not  a  student  of  logic 
unless  he  thinks  about  reasoning.  Now  reasoning  is  itself  a  special  kind  of  thinking; 
hence  the  special  kind  of  study  known  as  logic  is  concerned  with  thinking  about 
thinking.75 

Black  linked  the  study  of  logic  to  the  concept  of  critical  evaluation: 

Logic  may  be  thought  of  as  the  art  of  improving  reasoning,  and  the  science  of  the 
conditions  to  which  this  art  must  conform;  it  therefore  serves  the  practical  interest  of  self- 
betterment  and  the  theoretical  interest  of  understanding.  Logic  aims  at  the  cultivation  of 
the  art  and  science  of  the  criticism  of  reasoning."  Criticism"  is  intended  to  mean  here  the 
exhibition  of  standards  and  principles.  The  logician  pursues  certain  ideals,  and  wishes  to 
understand  the  objective  conditions  that  have  to  be  satisfied  if  those  ideals  are  to  be 
attained.76 

In  2009,  Brooke  Noel  Moore  and  Richard  Parker  published  the  ninth  edition  of  their  book, 
Critical  Thinking?1  As  with  Black’s  book,  this  textbook,  intended  for  a  critical  thinking  class,  is 
primarily  focused  on  logic,  with  a  strong  emphasis  on  syllogisms,  fallacies,  and  detennining  the 
validity  of  arguments. 

In  contrast  to  many  philosophers,  others  believe  that  logic  is  insufficient  for  an  understanding  of 
and  an  ability  to  apply  critical  thinking.  For  example,  John  McPeck  took  a  very  strong  stand 
against  logic  as  a  means  to  thinking  critically: 

In  the  present  chapter  I  shall  attempt  to  show  why  courses  in  logic  fail  to  accomplish  the 
goal  of  developing  critical  thinkers  and  how  the  epistemology  of  various  subjects  would 
be  the  most  reasonable  route  to  that  end.  Ironically,  as  it  turns  out,  the  epistemological 
approach  to  critical  thinking  involves  little  more  than  providing  what  has  always  been  a 
necessary  condition  of  education,  namely,  understanding  what  constitutes  good  reasons 
for  various  beliefs.  In  short,  there  is  both  a  conceptual  and  a  pedagogic  link  between 
epistemology,  critical  thinking  and  education,  but  the  study  of  logic  or  critical  thinking  as 

no 

such  has  no  part  in  this  linkage. 

David  Schum  and  Francis  Hume  discussed  critical  reasoning  within  the  context  of  intelligence 
analysis  in  The  Art  and  Science  of  The  Process  of  Intelligence  Analysis  and  stated  that  logic  is 
not  sufficient  for  the  kinds  of  tasks  an  analyst  must  perfonn. 

The  truth  is  that  analysts  are  routinely  asked  to  perfonn  tasks  for  which  they  have 
received  little  if  any  adequate  tutoring.  Conventional  courses  in  logic,  probability,  and 
statistics  do  not  prepare  a  person  for  the  task  of  drawing  conclusions  based  on  masses  of 


75  Black,  M.  Critical  Thinking:  An  Introduction  to  Logic  and  Scientific  Method,  2nd  ed.,  Prentice-Hall,  New  York, 
NY,  1952,  p.  3. 

76  Black,  pp.  9-10. 

77  Moore,  B.N.  &  Parker,  R.  Critical  Thinking,  9th  ed.,  McGraw-Hill,  Boston,  MA,  2009. 

78  McPeck,  J.E.  Critical  Thinking  and  Education,  St.  Martin's  Press,  New  York,  NY,  1981,  p.  22. 
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evidence  whose  items  suggest  many,  often  complex  and  interrelated,  lines  of  arguments 
on  hypotheses  of  interest.79 

Claudia  Maria  Alvarez  Ortiz  conducted  a  meta-analysis  of  studies  to  determine  whether  the 
study  of  philosophy  improves  critical  thinking  abilities. 

In  brief,  the  key  findings  of  the  meta-analysis  are  as  follows: 

•  First,  there  is  insufficient  evidence  to  be  confident  that  studying  philosophy 
improves  critical  thinking  skills  any  more  than  studying  other  academic  disciplines. 

•  Second,  the  results  indicate  that  studying  philosophy  appears  to  be  less  effective 

on 

than  studying  critical  thinking  in  its  own  right. 

If  we  don’t  have  a  clear,  shared  definition  of  critical  thinking  and  some  authors  argue  that  critical 
thinking  is  more  than  logic,  what  is  it? 

4.1.4.  Summary:  What  is  Critical  Thinking? 

Everyone  thinks  every  day,  but  what  distinguishes  critical  from  non-critical  thinking?  After 
reviewing  over  500  documents  about  critical  thinking  I  have  concluded  that  the  term  critical 
thinking  represents  a  concept  that  describes  a  special  quality  of  thinking.  Dewey’s  description  of 
critical  and  non-critical  reflection  helps  clarify  the  special  quality  of  thinking  we  now  call 
critical  thinking : 

For  the  person  may  not  be  sufficiently  critical  about  the  ideas  that  occur  to  him.  He  may 
jump  at  a  conclusion  without  weighing  the  grounds  on  which  it  rests;  he  may  forego  or 
unduly  shorten  the  act  of  hunting,  inquiring;  he  may  take  the  first  'answer,'  or  solution, 
that  comes  to  him  because  of  mental  sloth,  torpor,  impatience  to  get  something  settled. 
One  can  think  reflectively  only  when  one  is  willing  to  endure  suspense  and  to  undergo 
the  trouble  of  searching.  Too  many  persons  both  suspense  of  judgment  and  intellectual 
search  are  disagreeable;  they  want  to  get  them  ended  as  soon  as  possible.  They  cultivate 
an  over-positive  and  dogmatic  habit  of  mind,  or  feel  perhaps  that  a  condition  of  doubt 
will  be  regarded  as  evidence  of  mental  inferiority.  It  is  at  the  point  where  examination 
and  test  enter  into  investigation  that  the  difference  between  reflective  thought  and  bad 
thinking  comes  in.  To  be  genuinely  thoughtful,  we  must  be  willing  to  sustain  and 
protract  that  state,  of  doubt  which  is  the  stimulus  to  thorough  inquiry,  so  as  not  to  accept 
an  idea  or  make  positive  assertion  of  a  belief  until  justifying  reasons  have  been  found.81 

Critical  thinking  refers  to  how  one  approaches  the  thinking  task  as  well  as  what  activities  and 
skills  one  employs.  It  describes  the  act  of  thinking  critically,  thinking  with  a  critical  attitude, 
thinking  that  approaches  thinking  tasks  deliberately,  thoroughly,  objectively,  holistically,  and 
systematically.  Activities  and  traits  such  as  those  listed  in  Figure  2  are  hows  that  might  be 
especially  useful  in  contributing  to  the  quality  of  thinking  called  critical  thinking.  This 
description  includes  cognitive  skills  in  order  to  clarify  the  kinds  of  cognitive  activities  needed  in 
critical  thinking.  It  also  includes  descriptions  of  attitudes  that  contribute  to  the  critical  quality  of 
thinking. 


79  Hughes,  F.J.  &  Schum,  D.A.  The  Art  and  Science  of  the  Process  of  Intelligence  Analysis,  Joint  Military 
Intelligence  College,  Washington,  D.C.,  2003,  p.  2. 

so  Ortiz,  C.M.A.  Does  Philosophy  Improve  Critical  Thinking  Skills?  MA  Thesis,  University  of  Melbourne,  2007,  p. 
3. 

81  Dewey,  J.  How  We  Think:  A  Restatement  of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed., 
Regnery,  Chicago,  IL,  1933,  p.  16. 
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OJ 

This  description  of  critical  thinking  is  most  closely  linked  to  the  definition  provided  by  Kurfiss.  " 
The  list  of  cognitive  skills  is  most  closely  linked  to  those  included  in  the  definitions  from  the 
Foundation  of  Critical  Thinking,  the  APA  Consensus  definition,  the  description  of  thinking 

o  c 

critically  provided  by  Glaser,  and  the  description  of  critical  judgment  provided  by  the  US 
Educational  Policies  Commission.86  The  list  of  character  traits  is  most  closely  linked  with  the 
descriptions  provided  by  Dewey,  the  APA  Consensus  Definition,  and  Glaser. 


82  Kurfiss,  J.G.  Critical  Thinking:  Theory,  Research,  Practice,  and  Possibilities,  ASHE-ERIC  Higher  Education 
Report  No.  2,  Association  for  the  Study  of  Higher  Education,  Washington,  D.C.,  1988,  p.  2. 

83  Scriven,  M.  &  Paul,  R.  “Defining  Critical  Thinking,”  Foundation  for  Critical  Thinking.  Available: 
http://www.criticalthinking.org/page.cfm?PageID=410&CategorylD=51,  Viewed:  March  16,  2010. 

84  Facione,  P.A.  Critical  Thinking:  A  Statement  of  Expert  Consensus  for  Purposes  of  Educational  Assessment  and 
Instruction.  Research  Findings  and  Recommendations.  American  Philosophical  Association,  Newark,  DE,  1990,  p. 
3. 

85Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  pp. 
5-6. 

86  Educational  Policies  Commission.  The  Purposes  of  Education  in  American  Democracy,  National  Educational 
Association  of  the  United  States,  Washington,  D.C.,  1938,  p.  35. 

87  Dewey,  J.  How  We  Think:  A  Restatement  of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed., 
Regnery,  Chicago,  IL,  1933,  p.  16. 

88  Facione,  P.A.  Critical  Thinking:  A  Statement  of  Expert  Consensus  for  Purposes  of  Educational  Assessment  and 
Instruction.  Research  Findings  and  Recommendations.  American  Philosophical  Association,  Newark,  DE,  1990,  p. 
3. 

89  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  pp. 
5-6. 
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Critical  Thinking  Description 

Critical  thinking  is  a  quality  of  thinking  useful  when  exploring  a  situation,  question,  or  problem  to 
arrive  at  a  hypothesis  or  conclusion  about  it  that  integrates  all  available  infonnation  and  that  can 
be  convincingly  justified. 

Thinking  critically  requires  cognitive  dispositions  and  attitudes  such  as: 

•  Willingness  to  engage  in  intellectual  effort 

•  Systematic 

•  Persistent 

•  Intellectual  integrity 

•  Open-mindedness,  focusing  on  objectivity,  remaining  open  to  what  the  evidence  tells 
you 

Thinking  critically  also  depends  on  cognitive  skills  such  as: 

•  Interpretation  of  infonnation 

•  Analysis  of  information 

•  Evaluation  of  information 

•  Synthesis  of  information 

•  Inference  from  information 

•  Ability  to  explain  the  hypothesis  or  conclusion  and  the  evidence  that  led  to  the 
judgment 

Figure  2:  Recommended  Critical  Thinking  Description 

Although  this  description  appears  to  be  specific  and  perhaps  even  measurable,  critical  thinking 
remains  only  a  concept.  We  still  cannot  distinguish  whether  someone  is  thinking  critically  or 
not,  we  cannot  confirm  that  the  thinker  searched  sufficiently,  remained  in  suspense  long  enough, 
and  accurately  evaluated,  analyzed  or  synthesized  the  information  in  order  to  infer  the 
appropriate  answer. 

4.2  Enablers  of  Critical  Thinking:  Skills,  Dispositions,  and  Personal  Epistemology 

Now  that  we  have  established  a  potential  characterization  of  critical  thinking,  we  explore  the 
question  of  those  factors  that  enable  someone  to  think  critically. 

Many  critical  thinking  scholars  have  addressed  the  question  of  what  skills  are  needed  to  think 
critically.  Many  more  have  emphasized  the  importance  of  individual  characteristics,  typically 
called  thinking  dispositions,  because  they  believe  that  one  can  have  a  skill  related  to  critical 
thinking,  yet  not  use  the  skill  without  a  disposition  or  motivation  to  think  critically. 

In  addition  to  the  issues  surrounding  skills  and  dispositions,  another  factor  that  contributes  to  a 
person’s  critical  thinking  ability  is  his  or  her  personal  epistemology,  or  beliefs  about  knowledge 
and  knowing.  Educational  and  developmental  psychologists  researching  the  topic  of  personal 
epistemology  have  found  that  an  individual’s  beliefs  about  knowledge  and  knowing  have  an 
impact  on  a  person’s  ability  and  propensity  to  think  critically.  A  particular  set  of  epistemological 
beliefs  can  lead  one  person  to  accept  infonnation  without  assessment  while  contrasting 
epistemological  beliefs  can  lead  another  person  to  question  information  actively.  A  mature 
personal  epistemology  can  be  thought  of  as  the  foundation  for  dispositions  that  lead  to  critical 
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thinking.  Although  a  number  of  researchers  are  active  in  the  field  of  personal  epistemology,  the 
majority  of  authors  focused  on  critical  thinking  do  not  acknowledge  the  role  of  personal 
epistemology  in  their  work. 

This  section  of  the  report  explores  these  enablers  of  critical  thinking: 

•  Critical  thinking  skills  -  as  defined  by  scholars,  as  defined  for  intelligence  analysts, 
and  recommendations 

•  Critical  thinking  dispositions  -  as  defined  by  scholars,  as  defined  for  intelligence 
analysts,  and  recommendations 

•  Personal  epistemology  -  an  introduction  to  the  concept  of  personal  epistemology 
and  how  it  relates  to  critical  thinking 

4.2.1.  Critical  Thinking  Skills 

Dictionary.com  defines  skill  as  “the  ability,  coming  from  one's  knowledge,  practice,  aptitude, 
etc.,  to  do  something  well.”  Merriam-Webster.com  defines  skill  as  “a  learned  power  of  doing 
something  competently;  a  developed  aptitude  or  ability.”  A  thinking  skill,  therefore,  is  a 
cognitive  operation  that  one  can  and  should  learn  to  do  proficiently. 

The  description  of  critical  thinking  in  Figure  2  provides  a  useful  starting  point  for  identifying 
discrete  critical  thinking  skills: 

•  Interpretation 

•  Analysis 

•  Evaluation 

•  Synthesis 

•  Inference 

•  Ability  to  explain  a  hypothesis  or  conclusion,  as  well  as  the  evidence  that  led  to  the 
judgment 

One  challenge  in  understanding  and  evaluating  the  skills  proposed  by  different  scholars  is  the 
very  different  approaches  used  to  describe  the  skills.  Some  scholars  focused  on  individual 
actions  such  as  “Determine  the  reliability  of  a  source.”  Some  scholars  focused  on  the  outcome 
of  the  thinking  activity  such  as  “Present  a  coherent  and  persuasive  argument  on  a  controversial, 
contemporary  topic.”  Some  focused  on  high-level  cognitive  thinking  skills  such  as  inference, 
and  some  included  all  three  types. 

An  additional  difference  in  approach  to  describing  skills  is  how  well-rounded  the  lists  of  skills 
are.  Some  scholars  wrote  about  just  one  or  two  of  the  skill  areas  identified  in  Figure  2,  while 
others  provided  a  more  complete  view  of  skills. 

Scholarly  Lists  of  Critical  Thinking  Skills :  Edward  Glaser  provided  one  of  the  earliest  lists  of 
critical  thinking  skills  in  his  1941  report  on  An  Experiment  in  the  Development  of  Critical 
Thinking.  Glaser’s  list  is  fairly  well-rounded,  touching  on  multiple  aspects  of  critical  thinking, 
including  analysis,  evaluation,  inference,  interpretation,  and  synthesis.  His  list  of  critical 
thinking  skills  included  the  ability  to: 
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•  recognize  problems 

•  find  workable  means  for  meeting  those  problems 

•  gather  and  marshal  pertinent  information 

•  recognize  unstated  assumptions  and  values 

•  comprehend  and  use  language  with  accuracy,  clarity,  and  discrimination 

•  interpret  data 

•  appraise  evidence  and  evaluate  arguments 

•  recognize  the  existence  (or  non-existence)  of  logical  relationships  between 
propositions 

•  draw  warranted  conclusions  and  generalizations 

•  test  the  conclusions  and  generalizations  at  which  one  arrives 

•  reconstruct  one's  patterns  of  beliefs  on  the  basis  of  wider  experience 

•  render  accurate  judgments  about  specific  things  and  qualities  in  everyday  life  90 

Although  Glaser’s  list  is  comprehensive,  other  authors  rarely  cited  it. 

As  with  other  aspects  of  critical  thinking,  the  literature  reveals  diverse  perspectives  on  the 
importance  of  critical  thinking  skills  and  disagreement  on  which  of  these  skills  are  actually 
important.  Even  individual  scholars  changed  their  perspective  over  time.  For  example,  Robert 
Ennis  fundamentally  changed  his  list  of  skills  from  his  initial  1962  list  in  “A  Concept  of  Critical 
Thinking”  to  his  1987  article  called  “A  Taxonomy  of  Critical  Thinking  Dispositions  and 
Abilities.”  Table  3  shows  how  Ennis’  lists  of  skills  evolved. 


90  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,pp. 
5-6. 


31 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-2014-1598,  Cleared  11  April  2014 


Table  3:  Critical  Thinking  Skills  as  Defined  by  Ennis  Over  Time 


A  Concept  of  Critical  Thinking 91 

1 .  Grasping  the  meaning  of  a  statement. 

2.  Judging  whether  there  is  ambiguity  in  a  line 
of  reasoning. 

3.  Judging  whether  certain  statements 
contradict  each  other. 

4.  Judging  whether  a  conclusion  follows 
necessarily. 

5.  Judging  whether  a  statement  is  specific 
enough. 

6.  Judging  whether  a  statement  is  actually  the 
application  of  a  certain  principle. 

7.  Judging  whether  an  observation  statement  is 
reliable. 

8.  Judging  whether  an  inductive  conclusion  is 
warranted. 

9.  Judging  whether  the  problem  has  been 
identified. 

10.  Judging  whether  something  is  an  assumption. 

1 1 .  Judging  whether  a  definition  is  adequate. 

12.  Judging  whether  a  statement  made  by  an 
alleged  authority  is  acceptable. 


A  Taxonomy  of  Critical  Thinking 
Dispositions  and  Abilities 92 

1.  Focusing  on  a  question 

2.  Analyzing  arguments 

3.  Asking  and  answering  questions  of 
clarification  and/or  challenge 

4.  Judging  the  credibility  of  a  source 

5.  Observing  and  judging  observation 
reports;  criteria: 

6.  Deducing  and  judging  deductions 

7.  Inducing  and  judging  inductions 
(including  inference) 

8.  Making  value  judgments 

9.  Defining  terms,  and  judging  definitions  in 
three  dimensions 

10.  Identifying  assumptions 

1 1 .  Deciding  on  an  action 

12.  Interacting  with  others 


Table  4  illustrates  four  different  scholarly  approaches: 


•  Barry  Beyer  (educator)  included  many  individual  actions,  most  of  which  fit  into  the 
skill  category  of  evaluation. 

•  Alec  Fisher  and  Michael  Scriven  (philosophers)  strongly  emphasized  the  word 
critical  and  addressed  several  categories  of  critical  thinking  skills  (e.g. 
interpretation,  analysis)  but  missed  the  important  category  of  inference  and 
expanded  into  areas  beyond  critical  thinking  (e.g.  critical  knowledge). 

•  Diane  Halpern  (psychologist)  provided  actions  rather  than  skills  and  contained 
items  in  common  with  other  critical  thinking  lists  (e.g.  synthesis)  as  well  as  skills 
that  pertain  to  cognition  in  general  (e.g.  recall  and  learning). 

•  The  Foundation  for  Critical  Thinking  (education  and  philosophy)  also  addressed 
some  from  our  definition  (e.g.  interpretation  and  inference),  but  also  went  beyond  to 
address  the  context  in  which  thinking  takes  place  (e.g.  concepts,  points  of  view). 


91  Ennis,  R.H.  "A  Concept  of  Critical  Thinking,"  Harvard  Educational  Review,  Vol.  32,  No.  1,  1962,  p.  84. 

92  Ennis,  R.H.  "A  Taxonomy  of  Critical  Thinking  Dispositions  and  Abilities,"  in  Teaching  Thinking  Skills :  Theory > 
and  Practice,  eds.  J.B.  Baron  &  R.J.  Sternberg,  Freeman,  New  York,  NY,  1987,  pp.  12-15. 
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Table  4:  Comparison  of  Critical  Thinking  Skills  by  Different  Scholars 


Beyer  93 

1 .  Distinguishing 
between  verifiable 
facts  and  value 
claims. 

2.  Determining  the 
reliability  of  a  source. 

3.  Determining  the 
factual  accuracy  of  a 
statement 

4.  Distinguishing 
relevant  from 
irrelevant 

information,  claims 

or  reasons. 

5.  Detecting  bias. 

6.  Identifying  unstated 
assumptions. 

7.  Identifying 
ambiguous  or 
equivocal  claims  or 
arguments. 

8.  Recognizing  logical 
inconsistencies  or 
fallacies  in  a  line  of 
reasoning. 

9.  Distinguishing 
between  warranted  or 
unwarranted  claims. 

10.  Determining  the 
strength  of  an 
argument. 

Fisher  &  Scriven  94 

1.  Critical  Interpretation:  Critical 

Reading,  Listening,  Observing 

•  Level  1 .  Active  (or  deep) 
understanding 

•  Level  2.  Active  inquiry 

•  Level  3.  Active 
generalization 

•  Level  4.  Active  self¬ 
scrutiny 

2.  Critical  Communication: 

Critical  Writing,  Speaking  and 

Presenting 

3.  Critical  Knowledge 

•  The  Standard  English 
Vocabulary  For  Critical 
Interpretation  and 

Appraisal 

•  A  Minimum  Technical 
Vocabulary  From  Logic 

•  Knowledge  About  The 
'Missing  In  Action'  Areas 

4.  Critical  Technique 

•  The  Interpretation  Of 
Context 

•  The  Clarification  Of 
Meaning 

•  The  Analysis  Of 

Arguments 

•  The  Synthesis  Of 
Considerations 

Halpern  94 

•  Recognize  semantic  slanting  and  guilt  by 
association. 

•  Seek  out  contradictory  evidence. 

•  Use  the  metacognitive  knowledge  that  allows 
novices  to  monitor  their  own  performance  and  to 
decide  when  additional  help  is  needed. 

•  Make  risk:  benefit  assessments. 

•  Generate  a  reasoned  method  for  selecting  among 
several  possible  courses  of  actions. 

•  Give  reasons  for  choices  as  well  as  varying  the  style 
and  amount  of  detail  in  explanations  depending  on 
who  is  receiving  the  information. 

•  Recall  relevant  information  when  it  is  needed. 

•  Use  skills  for  learning  new  techniques  efficiently 
and  relating  new  knowledge  to  information  that  was 
previously  learned. 

•  Use  numerical  information  including  the  ability  to 
think  probabilistically  and  express  thoughts 
numerically. 

•  Understand  basic  research  principles. 

•  Demonstrate  an  advanced  ability  to  read  and  write 
complex  prose. 

•  Present  a  coherent  and  persuasive  argument  on  a 
controversial,  contemporary  topic. 

•  Provide  complex  instructions  in  language  that  is 
appropriate  for  the  audience. 

•  Use  matrices  and  other  diagrams  for  communication. 

•  Synthesize  information  from  a  variety  of  sources. 

•  Determine  credibility  and  use  this  information  in 
formulating  and  communicating  decisions. 

Foundation  for  Critical 
Thinking  96 

The  competencies 
focusing  on  the  elements 
of  reasoning  and  the 
universal  intellectual 
standards  as  they  relate  to 
the  elements: 

1.  Purposes,  goals,  and 
objectives 

2.  Questions, 
problems,  and  issues 

3.  Information,  data, 
evidence,  and 
experience 

4.  Inferences  and 
interpretations 

5.  Assumptions  & 
presuppositions 

6.  Concepts,  theories, 
principles, 
definitions,  laws,  & 
axioms 

7.  Implications  and 
consequences 

8.  Points  of  view  and 
frames  of  reference 
The  competency 
focusing  on  the 
universal  intellectual 
standards: 

9.  Assessing  thinking 

93  Beyer,  B.K.  "Critical  Thinking:  What  is  it?"  Social  Education,  Vol.  49,  No.  4,  1985,  p.  272. 

94  Fisher,  A.  &  Scriven,  M.  Critical  Thinking:  Its  Definition  and  Assessment,  Edgepress,  Point  Reyes,  CA,  1997,  pp.  94-113. 

94Halpern,  D.F.  Thought  and  Knowledge:  An  Introduction  to  Critical  Thinking,  Lawrence  Erlbaum  Associates,  Mahwah,  NJ,  2003,  pp.  14-15. 

96  Paul,  R.  &  Elder,  L.  A  Guide  for  Educators  to  Critical  Thinking  Competency  Standards,  Foundation  for  Critical  Thinking,  Dillon  Beach,  CA,  2006,  p.  14. 
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Table  5  summarizes  widely  used  critical  thinking  assessments  and  the  skills  each  evaluates. 
Reviewing  the  assessments  provides  insight  into  what  thinking  skills  the  authors  of  the 
assessments  considered  important  and  how  the  authors  categorized  the  skills. 


Table  5:  Critical  Thinking  Skills  as  Assessed  in  Primary  Critical  Thinking  Assessments 


Watson-GIaser 

Cornell  Critical 

The  Ennis-Weir  Critical 

CCTST  100 

Critical  Thinking 

Thinking  Test 98 

Thinking  Essay  Test 99 

Appraisal  II  97 

•  Analysis 

Level  X: 

•  Getting  the  point 

•  Interpretation 

•  Recognize 

•  Induction 

•  Seeing  the  reasons  and 

•  Inference 

Assumptions 

•  Deduction 

assumptions 

•  Evaluation 

•  Evaluate 

•  Observation 

•  Stating  one's  point 

•  Inductive 

Arguments 

•  Credibility 

•  Offering  good  reasons 

reasoning 

•  Draw  conclusions 

•  Assumption 

•  Seeing  other  possibilities 

•  Deductive 

Identification 

•  Responding  appropriately 

reasoning 

to  and/or  avoiding: 

Level  Z: 

+  Equivocation 

•  Induction 

+  Irrelevance 

•  Deduction 

+  Circularity 

•  Observation 

+  Reversal  of  an  "if- 

•  Credibility 

then"  (or  other 

•  Assumption 

+  conditional) 

Identification 

relationship 

•  Meaning 

+  The  straw  person 

fallacy 

+  Overgeneralization 

+  Excessive  skepticism 

+  Credibility  problems 

+  The  use  of  emotive 

language  to  persuade 

The  participants  in  the  1990  APA  study  led  by  Peter  Facione  identified  major  categories  of  skills 
with  related  sub-skills.  This  representation  of  critical  thinking  skills  overlaps  the  skills  in  our 
description  of  critical  thinking  as  presented  in  Figure  2.  Several  documents  quoted  these  skills, 
including  a  US  Department  of  Education’s  study101  about  critical  thinking.  Facione  also 
provided  this  caveat  summarizing  the  perspective  of  the  APA  study  participants  about  skills: 


97  Watson,  G.  &  Glaser,  E.M.  Watson-GIaser  II  Critical  Thinking  Appraisal  Technical  Manual  and  User's  Guide, 
NCS  Pearson,  Inc.,  San  Antonio,  TX,  2009,  p.2. 

98  Ennis,  R.H.,  Millman,  J.  &  Tomko,  T.N.  Cornell  Critical  Thinking  Tests  Level  X &  Level  Z  Manual,  5th  ed.,  The 
Critical  Thinking  Co.,  Seaside,  CA,  2005,  p.  2. 

99  Ennis,  R.H.  &  Weir,  E.  The  Ennis-Weir  Critical  Thinking  Essay  Test,  Midwest  Publications,  Pacific  Grove  CA, 
1985,  p.  1. 

100  “California  Critical  Thinking  Skills  Tests,”  Insight  Assessment.  Available:  http://www.insightassessment.com/. 
Viewed:  April  29,  2010. 

101  Jones,  E.A.,  Hoffman,  S.,  Moore,  L.M.,  Ratcliff,  G.,  Tibbetts,  S.  &  Click,  B.A.L.  National  Assessment  of  College 
Student  Learning:  Identifying  College  Graduates'  Essential  Skills  in  Writing,  Speech  and  Listening,  and  Critical 
Thinking.  Final  Project  Report,  National  Center  on  Post  Secondary  Teaching,  Learning,  and  Assessment,  University 
Park,  PA,  1995. 
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The  experts  caution  that  CT  skills  can  usefully  be  grouped  and  subclassified  in  a  number 
of  legitimate  ways.  Hence,  the  sub-classification  which  resulted  from  this  Delphi 
research  should  not  be  interpreted  as  necessarily  excluding  all  others.  Indeed,  while 
declaring  themselves  to  be  in  agreement  with  this  sub-classification,  various  participating 
experts  have  also  published  their  own  sub-classifications.102 

The  APA  study  identified  the  following  critical  thinking  skills  and  sub-skills: 

1 .  Interpretation  -  To  comprehend  and  express  the  meaning  or  significance  of  a  wide  variety 
of  experiences,  situations,  data,  events,  judgments,  conventions,  beliefs,  rules,  procedures 
or  criteria. 

•  Categorization 

•  Decoding  Significance 

•  Clarifying  Meaning 

2.  Analysis  -  To  identify  the  intended  and  actual  inferential  relationships  among  statements, 
questions,  concepts,  descriptions  or  other  forms  of  representation  intended  to  express 
beliefs,  judgments,  experiences,  reasons,  information,  or  opinions. 

•  Examining  Ideas 

•  Identifying  Arguments 

•  Analyzing  Arguments 

3.  Evaluation  -  To  assess  the  credibility  of  statements  or  other  representations  which  are 
accounts  or  descriptions  of  a  person's  perception,  experience,  situation,  judgment,  belief, 
or  opinion;  and  to  assess  the  logical  strength  of  the  actual  or  intend  inferential 
relationships  among  statements,  descriptions,  questions  or  other  forms  of  representation. 

•  Assessing  Claims 

•  Assessing  Arguments 

4.  Inference  -  To  identify  and  secure  elements  needed  to  draw  reasonable  conclusions;  to 
form  conjectures  and  hypotheses;  to  consider  relevant  information  and  to  educe  the 
consequences  flowing  from  data,  statements,  principles,  evidence,  judgments,  beliefs, 
opinions,  concepts,  descriptions,  questions,  or  other  forms  of  representation. 

•  Querying  Evidence 

•  Conjecturing  Alternatives 

•  Drawing  Conclusions 

5.  Explanation  -  To  state  the  results  of  one’s  reasoning;  to  justify  that  reasoning  in  terms  of 
the  evidential,  conceptual,  methodological,  criteriological  and  contextual  considerations 
upon  which  one's  results  were  based;  and  to  present  one’s  reasoning  in  the  form  of  cogent 
arguments. 

•  Stating  Results 

•  Justifying  Procedures 

•  Presenting  Arguments 


Facione,  p.  11. 
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6.  Self-Regulation  -  Self-consciously  to  monitor  one's  cognitive  activities,  the  elements  used 
in  those  activities,  and  the  results  educed,  particularly  by  applying  skills  in  analysis  and 
evaluation  to  one's  own  inferential  judgments  with  a  view  toward  questioning, 
confirming,  validating,  or  correcting  either  one's  reasoning  or  one's  results. 

•  Self  examination 

•  Self-correction  103 

Critical  Thinking  Skills  Defined  in  the  Intelligence  Analysis  Literature :  A  total  of  seven 
documents  identified  specific  skills  associated  with  critical  thinking  that  apply  to  intelligence 
analysis.  Although  overlaps  in  these  lists  exist,  a  common  understanding  of  the  skills  required  to 
think  critically  does  not  appear  to  exist  within  the  intelligence  community. 

Two  documents104  referenced  the  skills  from  the  APA  study  and  described  how  these  skills  relate 
to  intelligence  analysis. 

David  Moore’s  Critical  Thinking  and  Intelligence  Analysis  provided  a  synthesis  and 
interpretation  of  skills  as  defined  by  Glaser,  Halpem,  Fisher,  and  the  Foundation  for  Critical 
Thinking: 

Competencies  of  Critical  Thinkers 

•  Recognize  problems  or  questions  and  find  effective  means  of  solution 

•  Engage  in  meta-cognitive  activities  that  identify  assumptions,  biases,  and 
performance  as  solutions  are  developed 

•  Interpret  data,  appraise  evidence,  and  evaluate  statements  in  order  to  recognize 
logical  relationships  between  propositions 

•  Infer  warranted  conclusions  and  generalizations  from  evidence 

•  Test  generalizations  and  conclusions  by  seeking  out  contradictory  evidence  that 

enables  them  to  judge  the  credibility  of  claims 

•  Convey  sound,  well-reasoned  arguments 

•  Focus  on  the  process  of  reasoning  with  the  intention  of  improving  the  process105 

ICD  Number  610  contains  the  competency  directories  for  the  IC  workforce.  Two  of  the 
intelligence  analysis  documents106  that  listed  critical  thinking  skills  used  this  list.  The  skills  and 
sub-skills  identified  for  critical  thinking  for  non-supervisory  IC  employees  at  GS-15  and  below 
include: 

•  Logic 

•  Analysis 


Facione,  pp.  13-19. 

104  Tam,  C.K.  Behavioral  and  Psychosocial  Considerations  in  Intelligence  Analysis:  A  Preliminary  Review  of 
Literature  on  Critical  Thinking  Skills,  AFRL-RH-AZ-TR- 1009-0009,  Air  Force  Research  Laboratory,  Mesa,  A Z, 
2009  and  Fless,  J.  &  Friedel,  C.  "Applying  Critical  Thinking  to  Intelligence  Analysis,"  American  Intelligence 
Journal,  Vol.  26,  No.  1,  2008,  pp.  31-44. 

105  Moore,  D.T.  Critical  Thinking  and  Intelligence  Analysis,  Joint  Military  Intelligence  College,  Washington,  D.C., 
2006,  p.  14. 

106  Anonymous.  National  Defense  Intelligence  College.  Academic  Year  2008-2009,  National  Defense  Intelligence 
College,  Washington,  D.C.,  [undated]  and  Spracher,  W.C.  National  Security  Intelligence  Professional  Education:  A 
Map  ofU.  S.  Civilian  University  Programs  and  Competencies,  Ph.D.  dissertation,  George  Washington  University, 
2009.' 


36 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-2014-1598,  Cleared  11  April  2014 


•  Synthesis 

•  Evaluation 

•  Creativity,  creative  thinking 

•  Judgment 

•  Exploring  alternatives 

•  Enterprise  perspective 

•  Situational  awareness107 

The  skills  and  sub-skills  identified  for  supervisory  and  managerial  IC  employees  at  GS-15  and 
below  include: 

•  Logic 

•  Analysis 

•  Synthesis 

•  Evaluation 

•  Creativity 

•  Judgment 

•  Decisiveness 

•  Flexibility 

•  Problem  solving108 

Critical  Thinking  Skills  in  This  Report  I  reviewed  the  skills  listed  in  all  documents  related  to 
critical  thinking  and  compared  them  with  our  description  of  critical  thinking  in  Figure  2.  Asa 
result  of  this  comparison,  it  became  clear  that  the  broad  categories  of  skills  provided  in  our 
description,  combined  with  the  addition  of  self-regulation  from  the  APA  study,  provide  a  useful 
summary  of  the  major  skills  that  enable  intelligence  analysts  to  think  critically.  Therefore,  my 
recommended  list  of  critical  thinking  skills  is: 

•  Interpretation 

•  Analysis 

•  Evaluation 

•  Inference 

•  Synthesis 

•  Explanation 

•  Self-regulation 

Table  6  illustrates  how  a  selection  of  the  skills  identified  by  various  authors  map  to  these  skills. 


107  Office  of  the  Director  of  National  Intelligence.  Intelligence  Community  Directive  Number  610.  Competency 
Directories  for  the  Intelligence  Community  Workforce.  2008,  p.  B-2. 

10S  Office  of  the  Director  of  National  Intelligence,  pp.  C-l  -  C-2. 
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Table  6:  Recommended  Critical  Thinking  Skills  for  Intelligence  Analysis  Mapped  to  Skills 


Identified  by  Scholars 


Interpretation 

Analysis 

Evaluation 

Inference 

+  Categorization 
+  Decoding  Significance 

+  Clarifying  Meaning 

+  Interpret  data 
+  Comprehend  language 
with  accuracy,  clarity, 
and  discrimination 
+  Grasping  the  meaning 
of  a  statement 
+  Judging  whether  a 
statement  is  over 
vague  or  over  specific 
+  Interpreting 
information 

+  Asking  and  answering 
questions  of 
clarification  and/or 
challenge 

+  Observing  and  judging 
observation  reports; 
criteria 

+  Critical  interpretation: 
critical  reading, 
listening,  observing 
+  The  interpretation  of 
context 

+  Examining  Ideas 

+  Identifying  Arguments 

+  Analyzing  Arguments 

+  Recognize  the 

existence  (or  non¬ 
existence)  of  logical 
relationships  between 
propositions, 

+  Recognizing 

organizing  techniques 
or  principles 
+  Analysis  of  related 
parts  of  a  problem 
+  Analysis  of  arguments 
+  Recognizing  logical 
inconsistencies  or 
fallacies 

+  Assessing  Claims 

+  Assessing  Arguments 

+  Appraise  evidence  and 
evaluate  arguments 
+  Identifying  unstated 
assumptions  and 
values 

+  Distinguishing 
between  verifiable 
facts  and  value  claims. 

+  Evaluating  sources  of 
information 

+  Distinguishing  relevant 
from  irrelevant 
information,  claims  or 

reasons. 

+  Detecting  bias 
+  Distinguishing 

between  warranted  or 
unwarranted  claims. 

+  Recognizing  the 
adequacy  of  data 
+  Recognizing 

persuading  techniques 

+  Querying  Evidence 

+  Conjecturing 
Alternatives 

+  Drawing  Conclusions 

+  Draw  warranted 
conclusions  and 
generalizations 

+  Render  accurate 
judgments  about 
specific  things  and 
qualities  in  everyday 
life 

+  Deducing  and  judging 
deductions 

+  Inducing  and  judging 
inductions 

+  Judging  whether  a 
conclusion  follows,  is 
warranted 

+  Distinguish  hypothesis 
from  evidence 

+  Distinguish  hypothesis 
from  warranted 
conclusion 

+  Construct  arguments 
supporting  judgment 

Explanation 

Self-Regulation 

Synthesis 

+  Stating  Results 
+  Justifying  Procedures 
+  Presenting  Arguments 
+  Use  language  with 
accuracy,  clarity,  and 
discrimination 
+  Interacting  with  others 
+  Critical 

Communication: 

Critical  Writing, 
Speaking  and 

Presenting 
+  Demonstrate  an 
advanced  ability  to 
read  and  write 
complex  prose. 

+  Present  a  coherent  and 
persuasive  argument 
on  a  controversial, 
contemporary  topic. 

+  Use  matrices  and  other 
diagrams  for 
communication. 

+  Self  examination 

+  Self-correction 

+  Test  the  conclusions 

and  generalizations  at 
which  one  arrives 
+  Refraining  from 

jumping  to  conclusion 
+  Retention  of  an  open 

mind. 

+  Be  aware  of  own  point 

of  view  and  biases  this 
may  cause 

+  Revise  argument  in 

light  of  new  evidence 
+  Seek  out  contradictory 

evidence. 

+  Use  the  metacognitive 

knowledge  that  allows 
novices  to  monitor 
their  own  performance 
and  to  decide  when 
additional  help  is 
needed. 

+  The  synthesis  of 
considerations 
+  Synthesize  information 
from  a  variety  of 
sources. 

+  Synthesis 
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4.2.2.  Critical  Thinking  Dispositions 

The  ability  to  employ  critical  thinking  skills  is  still  not  sufficient  to  ensure  that  an  individual  will 
bring  a  critical  approach  to  thinking.  A  thinker  must  be  willing  to  make  the  extra  effort  to  think 
critically.  The  thinker  must  have  the  disposition  to  think  critically. 

The  concern  with  thinking  dispositions  was  even  more  popular  in  the  literature  than  critical 
thinking  skills.  Fifty  four  documents  addressed  the  topic  of  skills  while  71  addressed  thinking 
dispositions.  Just  as  with  critical  thinking  skills,  the  specific  dispositions  associated  with  critical 
thinking  have  varied  over  the  years  and  between  authors. 

Merriam-Webster.com  defines  disposition  as,  “prevailing  tendency,  mood,  or  inclination.” 
Dictionary.com  defines  disposition  as,  “state  of  mind  regarding  something;  inclination.”  It  is 
interesting  to  note  that  the  majority  of  scholars  who  wrote  about  dispositions  described  them  as 
something  that  could  be  taught  and  developed,  similar  to  teaching  and  developing  skills.  For 
example,  Barry  Beyer  noted: 

Dispositions  such  as  those  noted  here  do  not  necessarily  develop  on  their  own.  They,  too, 
must  be  taught.  These  dispositions  are  not,  however,  taught  in  exactly  the  same  ways  as 
are  the  skilled  operations  of  thinking.  Teaching  them  takes  much  longer  than  does 
teaching  a  specific  skill  and  requires  use  of  direct  as  well  as  indirect  teaching  techniques 
and  strategies.109 

Our  description  of  critical  thinking  in  Figure  2  included  five  dispositional  characteristics: 

•  Willingness  to  engage  in  intellectual  effort 

•  Systematic 

•  Persistent 

•  Intellectual  integrity 

•  Open-mindedness,  a  focus  on  objectivity,  remaining  open  to  what  the  evidence 
tells  you 

The  earliest  critical  thinking  scholars  highlighted  the  importance  of  a  critical  attitude.  In  How 
We  Think,  John  Dewey  noted  that  a  thinker  needs  more  than  skills.  He  identified  three  primary 
dispositions:  open  mindedness,  whole  heartedness,  and  responsibility: 

Knowledge  of  the  methods  alone  will  not  suffice;  there  must  be  the  desire,  the  will,  to 
employ  them.  This  desire  is  an  affair  of  personal  disposition.  But  on  the  other  hand  the 
disposition  alone  will  not  suffice.  There  must  also  be  understanding  of  the  forms  and 
techniques  that  are  the  channels  through  which  these  attitudes  operate  to  the  best 
advantage.110 

Glaser  also  identified  the  importance  of  the  thinker’s  attitude.  His  definition  of  critical  thinking 
starts  with  an  emphasis  on  the  importance  of  attitude: 


109  Beyer,  B.K.  Practical  Strategies  for  the  Teaching  of  Thinking,  Allyn  and  Bacon,  Boston,  MA,  1987,  p.  21 1. 

1 10  Dewey,  J.  How  We  Think:  A  Restatement  of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed., 
Regnery,  Chicago,  IL,  1933,  p.  30  . 
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The  ability  to  think  critically,  as  conceived  in  this  volume,  involves  three  things:  (1)  an 
attitude  of  being  disposed  to  consider  in  a  thoughtful  way  the  problems  and  subjects  that 
come  within  the  range  of  one's  experiences,111 

He  followed  his  definition  with  this  explanation: 

The  development  of  ability  to  think  critically,  it  should  be  noted,  is  not  limited  to 
cultivation  of  better  methods  for  finding  and  testing  evidence  and  meanings,  and  arriving 
at  well-founded  conclusions.  Knowledge  of  the  methods  of  logical  inquiry  is  important. 
Even  more  important  for  the  everyday  practice  of  democracy,  however,  are  the  attitudes 
involved  in  critical  thinking.  Persons  who  have  acquired  a  disposition  to  want  evidence 
for  beliefs,  and  who  have  acquired  an  attitude  of  reasonableness  have  also  acquired 
something  of  a  way  of  life  which  makes  for  more  considerate  and  humane  relationships 
among  men.112 

Not  all  authors  stated  a  belief  in  the  importance  of  dispositions.  Michael  Fisher  and  Alec  Scriven 
provided  a  contrary  view: 

A  deeper  issue  comes  up  here,  too,  about  the  affective  (values)  dimension.  We’ll  go  into 
it  a  little,  because  it  is  a  key  issue  about  the  nature  of  critical  thinking,  and  because 
Facione  was  awarded  a  big  federal  contract  for  further  work  on  critical  thinking  (1995), 
so  his  view  is  likely  to  become  more  influential.  Facione  supports  the  inclusion  of  (ten) 
affective  variables,  while  reporting  that  one  third  of  his  Delphi  group  -  this  was  the 
largest  dissident  group  on  any  topic  -  supported  the  view  expressed  in  this  chapter,  that 
critical  thinking  should  be  defined  simply  as  a  skilled  cognitive  process.  Unfortunately, 
the  dissident  minority  did  not  provide  a  set  of  items  that  refer  to  what  he  describes  as 
their  view.  They  may  have  been  rejecting  what  we  have  called  the  'active'  requirement, 
not  the  inclusion  of  affective  variables.113 

Scholarly  Lists  of  Critical  Thinking  Dispositions:  As  with  skills,  a  single  author  may  use 
different  language  at  different  times  to  describe  dispositions.  For  example,  Peter  Facione,  the 
author  of  the  APA  report  on  critical  thinking,  described  the  ideal  critical  thinker  and  also 
provided  a  list  of  critical  thinking  dispositions  in  the  same  report.  Overlap  exists  between  the 
two  lists,  but  they  are  not  consistent.  When  Facione  wrote  about  the  CCTDI  in  later 
publications,  he  described  broad  categories  of  dispositions  rather  than  the  list  of  dispositions  in 
the  APA  study.  Facione’s  refinement  of  dispositions  is  shown  in  Table  7. 


111  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  p. 
5. 

112  Glaser,  p.  6. 

113  Fisher,  A.  &  Scriven,  M.  Critical  Thinking:  Its  Definition  and  Assessment,  Edgepress,  Point  Reyes,  CA,  1997,  p. 
81. 
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Table  7:  Critical  Thinking  Dispositions  as  Defined  by  Facione  Over  Time 


APA/Facione 

The  ideal  critical  thinker  is 

•  habitually  inquisitive 

•  well  informed 

•  trustful  of  reason 

•  open-minded 

•  flexible 

•  fair-minded  in  evaluation 

•  honest  in  facing  personal 
biases 

•  prudent  in  making 
judgments 

•  willing  to  reconsider 

•  clear  about  issues 

•  orderly  in  complex  matters 

•  diligent  in  seeking  relevant 
information 

•  reasonable  in  the  selection 
of  criteria 

•  focused  in  inquiry 

•  persistent  in  seeking  results 
which  are  as  precise  as  the 
subject  and  the 
circumstances  of  inquiry 
permit. 


APA/Facione 

Affective  Dispositions  of  Critical  Thinking 

Approaches  to  life  and  living  in  general 

•  clarity  in  stating  the  question  or  concern, 

•  inquisitiveness  with  regard  to  a  wide  range  of 
issues, 

•  concern  to  become  and  remain  generally 
well-informed, 

•  alertness  to  opportunities  to  use  CT, 

•  trust  in  the  processes  of  reasoned  inquiry, 

•  self-confidence  in  one's  own  ability  to 
reason, 

•  open-mindedness  regarding  divergent  world 
views, 

•  flexibility  in  considering  alternatives  and 
opinions, 

•  understanding  of  the  opinions  of  other 
people, 

•  fair-mindedness  in  appraising  reasoning, 

•  honesty  in  facing  one's  own  biases, 
prejudices,  stereotypes,  egocentric  or 
sociocentric  tendencies, 

•  prudence  in  suspending,  making  or  altering 
judgments, 

•  willingness  to  reconsider  and  revise  views 
where  honest  reflection  suggests  that  change 
is  warranted. 


Facione 

CCTDI  assess  for: 

•  truthseeking, 

•  openmindedness, 

•  analytic  ity, 

•  systematic  ity, 

•  critical  thinking  self- 
confidence, 

•  inquisitiveness, 

•  maturity  of  judgment 


Approaches  to  specific  issues,  questions,  or 

problems: 

•  clarity  in  stating  the  question  or  concern, 

•  orderliness  in  working  with  complexity, 

•  diligence  in  seeking  relevant  information, 

•  reasonableness  in  selecting  and  applying 
criteria, 

•  care  in  focusing  attention  on  the  concern  at 
hand, 

•  persistence  though  difficulties  are 
encountered, 

•  precision  to  the  degree  permitted  by  subject 
and  circumstances. 


114  Facione,  P.A.  Critical  Thinking:  A  Statement  of  Expert  Consensus  for  Purposes  of  Educational  Assessment  and 
Instruction.  Research  Findings  and  Recommendations.  American  Philosophical  Association,  Newark,  DE,  1990,  p. 
3. 

115  Facione,  p.  25. 

116  Facione,  P.A.,  Facione,  N.C.  &  Giancarlo,  C.A.  "The  Disposition  Toward  Critical  Thinking:  Its  Character, 
Measurement,  and  Relationship  to  Critical  Thinking  Skill,"  Informal  Logic,  Vol.  20,  No.  1,  2000,  p.74. 
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Table  8  provides  a  comparison  of  dispositions  from  the  scholars  who  provided  the  lists  of  skills 
in  Table  4. 


•  Robert  Ennis  (philosopher  and  educator) 

•  Barry  Beyer  (educator) 

•  Diane  Halpern  (psychologist) 

•  Foundation  for  Critical  Thinking  (education  and  philosophy) 

Alec  Fisher  and  Michael  Scriven  (philosophers)  do  not  believe  dispositions  are  as  important  as 
skills  and  did  not  identify  any.  Therefore,  they  are  not  included  in  the  table. 

To  replace  Fisher  and  Scriven,  I  included  dispositions  defined  by  Keith  Stanovich,  a 
psychologist,  in  Table  8.  Stanovich  has  written  extensively  on  the  impact  that  thinking 
dispositions  have  on  reasoning  ability.117 


117  For  example  see  Stanovich,  K.E.  Who  is  Rational?  Studies  of  Individual  Differences  in  Reasoning,  L.  Erlbaum 
Associates,  Mahwah,  NJ,  1999,  Stanovich,  K.E.  &  West,  R.F.  "Individual  Differences  in  Rational  Thought," 
Journal  of  Experimental  Psychology’:  General,  Vol.  127,  No.  2,  1998,  pp.  161-188,  Stanovich,  K.E.  What 
Intelligence  Tests  Miss:  The  Psychology’  of  Rational  Thought,  Y  ale  University  Press,  New  Haven,  CT,  2009. 
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Table  8:  Comparison  of  Critical  Thinking  Dispositions  by  Different  Scholars 


Ennis  118 

1 .  Seek  a  clear  statement  of  the  thesis  or  question 

2.  Seek  reasons 

3.  Try  to  be  well  informed 

4.  Use  and  mention  credible  sources 

5.  Take  into  account  the  total  situation 

6.  Try  to  remain  relevant  to  the  main  point 

7.  Keep  in  mind  the  original  and/or  basic  concern 

8.  Took  for  alternatives 

9.  Be  open-minded 

a)  Consider  seriously  other  points  of  view  than 
one's  own  (dialogical  thinking) 

b)  Reason  from  premises  with  which  one 
disagrees — without  letting  the  disagreement 
interfere  with  one's  reasoning  (suppositional 
thinking) 

c)  Withhold  judgment  when  the  evidence  and 
reasons  are  insufficient 

10.  Take  a  position  (and  change  a  position)  when 
the  evidence  and  reasons  are  sufficient  to  do  so 

11.  Seek  as  much  precision  as  the  subject  permits 

12.  Deal  in  an  orderly  manner  with  the  parts  of  a 
complex  whole 

13.  Use  one's  critical  thinking  abilities 

14.  Be  sensitive  to  the  feelings,  level  of  knowledge, 
and  degree  of  sophistication  of  others 

Beyer119 

1 .  Seek  a  clear  statement  of  a 
problem,  a  thesis,  a  question. 

2.  Deliberately  examine  a  variety 
of  viewpoints. 

3.  Seek  to  be  well  informed. 

4.  Use  credible  sources. 

5.  Seek  a  number  of  alternatives. 

6.  Seek/give  reasons. 

7.  Seek/provide  evidence. 

8.  Be  open-minded. 

9.  Be  willing  to  change  a 
position/judgment  when 
evidence  and  reasons  are 
sufficient  to  do  so. 

10.  Judge  in  terms  of  situations, 
issues,  purposes,  and 
consequences  (not  in  terms  of 
fixed,  dogmatic  precepts  or 
emotional,  wistful  thinking). 

1 1 .  Persist  in  carrying  out  a 
thinking  task. 

12.  Be  slow  to  believe — be 
skeptical. 

13.  Be  objective. 

14.  Suspend  judgment  when 
appropriate/sufficient  evidence 
and  reasoning  are  lacking. 

Halpern120 

1.  Willingness  to 

Plan 

2.  Flexibility 

3.  Persistence 

4.  Willingness  to 
Self-Correct,  Admit 
Errors,  and  Change 
Your  Mind  When 
the  Evidence 

Changes 

5.  Being  Mindful 

6.  Consensus 

Seeking 

Foundation121 

•  Fairmindedne 

ss 

•  Intellectual 
Humility 

•  Intellectual 
Courage 

•  Intellectual 
Empathy 

•  Intellectual 
Integrity 

•  Intellectual 
Perseverance 

•  Confidence  in 
Reason 

•  Intellectual 
Autonomy 

Stanovich122 

•  the  tendency  to  collect 
information  before  making 
up  one's  mind, 

•  the  tendency  to  seek  various 
points  of  view  before 
coming  to  a  conclusion, 

•  the  disposition  to  think 
extensively  about  a  problem 
before  responding, 

•  the  tendency  to  calibrate  the 
degree  of  strength  of  one's 
opinion  to  the  degree  of 
evidence  available, 

•  the  tendency  to  think  about 
future  consequences  before 
taking  action, 

•  the  tendency  to  explicitly 
weigh  pluses  and  minuses  of 
situations  before  making  a 
decision, 

•  and  the  tendency  to  seek 
nuance  and  avoid 
absolutism 

118  Ennis,  R.H.  "A  Taxonomy  of  Critical  Thinking  Dispositions  and  Abilities,"  in  Teaching  Thinking  Skills:  Theory  and  Practice,  eds.  J.B.  Baron  &  R.  J. 
Sternberg,  Freeman,  New  York,  NY,  1987,  p.  12. 

119  Beyer,  B.K.  Practical  Strategies  for  the  Teaching  of  Thinking,  Allyn  and  Bacon,  Boston,  MA,  1987,  p.  212. 

120  Halpern,  D.F.  Thought  and  Knowledge:  An  Introduction  to  Critical  Thinking,  Fawrence  Erlbaum  Associates,  Mahwah,  NJ,  2003,  pp.  15-17. 

121  Paul,  R.  &  Elder,  F.  A  Guide  for  Educators  to  Critical  Thinking  Competency  Standards,  Foundation  for  Critical  Thinking,  Dillon  Beach,  CA,  2006,  pp.  14-15. 

122  Stanovich,  K.E.  What  Intelligence  Tests  Miss:  The  Psychology’  of  Rational  Thought,  Y  ale  University  Press,  New  Flaven,  CT,  2009,  pp.  31-32. 
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Critical  Thinking  Dispositions  in  the  Intelligence  Analysis  Literature:  Only  one  document  in 
the  intelligence  analysis  literature  discussed  critical  thinking  dispositions.  The  lack  of  attention 
regarding  thinking  dispositions  in  the  intelligence  literature  may  represent  an  omission  within  the 
intelligence  community’s  body  of  knowledge. 

David  Moore  in  Critical  Thinking  and  Intelligence  Analysis  referenced  the  Facione  umbrella 
dispositions  from  the  CCTDI  (listed  in  Table  7)  and  a  list  of  dispositions  defined  by  Richard  Paul 
and  Gerald  Nosich,  both  associated  with  the  Foundation  for  Critical  Thinking. 

According  to  Richard  Paul  and  Gerald  Nosich,  the  characteristics  of  critical  thinkers 
include  thinking  independently,  exercising  fair-mindedness,  developing  insight  into 
egocentricity  and  sociocentricity,  developing  intellectual  humility  and  suspending 
judgment,  developing  intellectual  courage,  developing  intellectual  good  faith  and 
integrity,  developing  intellectual  perseverance,  developing  confidence  in  reason, 
exploring  thoughts  underlying  feelings  and  feelings  underlying  thoughts,  developing 

1  9T 

intellectual  curiosity. 

Moore  highlighted  the  connection  between  the  dispositions  as  defined  by  Facione,  Paul,  and 
Nosich  and  the  characteristics  of  successful  intelligence  analysts  as  defined  by  Krizan  and 
Moore. 

Both  sets  closely  match  the  characteristics  of  successful  intelligence  analysts  identified 
by  Lisa  Krizan  and  the  author  in  their  work  on  intelligence  analysts'  core  competencies. 
Krizan  and  the  author  observe  that  successful  intelligence  analysts  are  insatiably  curious. 
Fascinated  by  puzzles,  their  high  levels  of  self-motivation  lead  them  to  observe  and  read 
voraciously,  and  to  take  fair-minded  and  varied  perspectives.  This  helps  them  to  make 
the  creative  connections  necessary  for  solving  the  hardest  intelligence  problems.  Finally, 
the  emotional  tensions  created  by  problems,  and  the  cathartic  release  at  their  solution, 
powerfully  motivate  analysts.124 

Several  documents  that  address  analyst  competencies  emphasize  characteristics  that  are  similar 
to  those  found  in  lists  of  thinking  dispositions.  For  example,  Lahneman’s  report  on  the  future  of 
intelligence  analysis  included  the  following  traits  of  successful  analysts.  Those  relevant  to 
critical  thinking  are  highlighted  in  blue: 


Traits 

o 

Natural  curiosity 

o 

Well-rounded 

o 

Clear  thinker 

o 

Critical  thinker 

o 

Intuitive  thinker 

o 

Team  player 

o 

“Sees  beyond  the  obvious' 

o 

Grasp  of  the  theoretical 

123  Moore,  D.T.  Critical  Thinking  and  Intelligence  Analysis,  Joint  Military  Intelligence  College,  Washington,  D.C., 
2006,  pp.  15-16. 

124  Moore,  D.T.  &  Krizan,  L.  "Intelligence  Analysis:  Does  NSA  Have  What  it  Takes?"  Cryptologic  Quarterly,  Vol. 
20,  No.  1-2,  2001,  p.  16. 
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o  Self-discipline 

o  Ability  to  handle  substantive  uncertainty 
o  Ability  to  address  alternative  scenarios 
o  Ethics/integrity 

o  Flexibility/adaptability 

o  Fast  learner 

o  Imagination 

I  'y  r 

o  Appreciate  own  biases/perspective 

Critical  Thinking  Dispositions  in  This  Report:  I  reviewed  the  dispositions  listed  in  all 
documents  related  to  critical  thinking  and  compared  them  with  our  description  of  critical 
thinking  in  Figure  2.  Asa  result  of  this  comparison,  it  became  clear  that  the  broad  categories  of 
dispositions  provided  in  our  description,  combined  with  the  addition  of  intellectual  curiosity 
provide  a  useful  summary  of  the  major  dispositions  that  enable  and  trigger  someone  to  think 
critically. 

While  we  have  long  had  an  understanding  of  common  cognitive  skills  such  as  analysis, 
synthesis,  and  evaluation,  the  same  is  not  true  for  dispositions.  There  is  great  overlap  in  the 
concepts  expressed  within  these  examples  of  thinking  dispositions,  but  the  language  used  to 
describe  the  disposition  varied.  For  example:  Ennis  identified  the  disposition  to  seek  reasons', 
Moore  identified  the  disposition  to  be  intellectually  curious',  Stanovich  identified  the  tendency  to 
collect  information',  and  Facione  identified  the  disposition  of  inquisitiveness.  All  appear  to 
describe  the  same  inclination  toward  gathering  information  before  coming  to  a  decision  or 
conclusion.  As  a  result,  it  is  very  difficult  to  say  definitively  that  one  term  is  more  appropriate 
than  another.  The  terms  I  chose  to  describe  these  dispositions  represent  terms  that  I  believe  to  be 
most  easily  understandable. 

Therefore,  my  recommended  list  of  critical  thinking  dispositions  is: 

•  Willingness  to  engage  in  intellectual  effort 

•  Systematic 

•  Persistent 

•  Intellectual  integrity 

•  Open-minded 

•  Intellectual  curiosity 

Table  9  illustrates  how  thinking  dispositions  identified  by  different  authors  fit  into  these 
categories. 


125  Lahneman,  W.J.  The  Future  of  Intelligence  Analysis.  Volume  1.  Final  Report.  Center  for  International  and 
Security  Studies  at  Maryland,  College  Park,  MD,  2006,  p.  A-l. 
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Table  9:  Recommended  Critical  Thinking  Dispositions  Mapped  to  Dispositions  Identified 


by  Scholars 


Willingness  to  Engage  Intellectually 

Systematic 

Persistent 

•  Alertness  to  opportunities  to  use  CT 

•  Trust  in  the  processes  of  reasoned 
inquiry 

•  Use  one’s  critical  thinking  abilities 

•  Intellectual  courage 

•  Being  mindful 

•  Confidence  in  reason 

•  Use  one's  critical  thinking  abilities 

•  Self-confidence  in  one's  own  ability 
to  reason 

•  Ability  to  handle  substantive 
uncertainty 

•  Sees  beyond  the  obvious 

•  The  disposition  to  think  extensively 
about  a  problem  before  responding 

•  The  tendency  to  seek  nuance  and 
avoid  absolutism 

•  Willingness  to  Plan 

•  Take  into  account  the  total 
situation 

•  Seek  a  clear  statement  of  the 
thesis  or  question 

•  Try  to  remain  relevant  to  the  main 
point 

•  Deal  in  an  orderly  manner  with 
the  parts  of  a  complex  whole 

•  Seek  as  much  precision  as  the 
subject  permits 

•  Clarity  in  stating  the  question  or 
concern,  orderliness  in  working 
with  complexity 

•  Care  in  focusing  attention  on  the 
concern  at  hand, 

•  Diligence  in  seeking  relevant 
information 

•  Intellectual  Perseverance 

•  Self  discipline 

•  Persistent  in  seeking 
results  which  are  as 
precise  as  the  subject  and 
the  circumstances  of 
inquiry  permit. 

•  Persistence  though 
difficulties  are 
encountered, 

•  Persist  in  carrying  out  a 
thinking  task. 

Open-Minded 

Intellectual  Integrity 

Intellectual  Curiosity 

•  Objectivity 

•  Actively  open-minded  thinking 

•  Consider  seriously  other  points  of 
view  than  one's  own 

•  Reason  from  premises  with  which 
one  disagrees — without  letting  the 
disagreement  interfere  with  one's 
reasoning 

•  Deliberately  examine  a  variety  of 
viewpoints. 

•  Fairmindedness 

•  Open-mindedness  regarding 
divergent  world  views 

•  Flexibility  in  considering 
alternatives  and  opinions 

•  Understanding  of  the  opinions  of 
other  people 

•  Fair-mindedness  in  appraising 
reasoning 

•  Appreciate  own  biases/perspective 

•  Ability  to  address  alternative 
scenarios 

•  Willingness  to  self-correct,  admit 
errors,  and  change  your  mind 
when  the  evidence  changes 

•  Be  willing  to  change  a 
position/judgment  when  evidence 
and  reasons  are  sufficient  to  do  so. 

•  Withhold  j  udgment  when  the 
evidence  and  reasons  are 
insufficient 

•  Intellectual  Autonomy 

•  The  tendency  to  calibrate  the 
degree  of  strength  of  one's  opinion 
to  the  degree  of  evidence  available 

•  Prudence  in  suspending,  making 
or  altering  judgments, 

•  Ethics/integrity 

•  Truthseeking 

•  Honesty  in  facing  one's  own 
biases,  prejudices,  stereotypes, 
egocentric  or  sociocentric 
tendencies 

•  Reason  from  premises  with  which 
one  disagrees — without  letting  the 
disagreement  interfere  with  one's 
reasoning  (suppositional  thinking) 

•  Try  to  be  well  informed 

•  Inquisitiveness  with 
regard  to  a  wide  range  of 
issues, 

•  Concern  to  become  and 
remain  generally  well- 
informed 

•  Natural  curiosity 

•  Deliberately  examine  a 
variety  of  viewpoints. 

•  Seek  to  be  well 
informed. 

•  Use  credible  sources. 

•  Seek  a  number  of 
alternatives. 

•  Seek/give  reasons. 

•  Seek/provide  evidence. 
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4.2.3.  Personal  Epistemology 

The  third  enabler  of  critical  thinking  is  personal  epistemology.  Researchers  in  personal 
epistemology  propose  that  one’s  personal  beliefs  about  knowledge  and  knowing  have  a 
fundamental  impact  on  the  disposition  and  ability  to  think  critically.  Although  researchers 
describe  personal  epistemology  in  different  ways  (e.g.  epistemic  orientation,  epistemological 
beliefs,  reflective  judgment)  and  evaluate  individual  perspectives  in  different  ways  (e.g.  levels  of 
understanding,  categories  of  beliefs,  developmental  stages),  a  common  theme  is  that  an 
individual  is  more  likely  to  engage  in  critical  thinking  when  his  or  her  personal  epistemology 
supports  a  complex  understanding  of  knowledge. 

A  mature  personal  epistemology  can  be  thought  of  as  the  foundation  for  dispositions  that  lead  to 
critical  thinking. 

The  concept  of  personal  epistemology  is  much  broader  than  can  be  explained  in  this  report.  A 
separate  report  expanding  on  personal  epistemology  will  be  available  in  the  future.  However,  a 
brief  introduction  to  the  concept  of  personal  epistemology  illustrates  its  potential  impact  on  an 
analyst’s  willingness  to  think  critically. 

Barbara  Hofer  linked  personal  epistemology  to  critical  thinking: 

Monitoring  our  own  understanding  of  the  complexity  of  problems,  the  certainty  and 
limits  of  knowledge,  and  the  evaluation  of  evidence  enables  the  critical  thinking 
necessary  to  solve  the  most  pressing  problems  we  may  face  as  individuals  and  as  a 
society.  The  importance  of  epistemic  assumptions  in  the  solving  of  ill-structured 
problems  is  another  critical  tie  to  the  work  of  cognitive  psychologists  (Voss  and  Post, 
1988),  whose  investigations  into  this  area  suggest  the  importance  of  such  tasks  in  human 
cognition.  Yet  the  evidence  gathered  to  date  suggests  that  most  individuals  do  not 
achieve  a  level  of  epistemological  understanding  that  makes  possible  genuine  critical 

thinking  (Kuhn,  1999a)  or  a  level  of  reflective  judgment  essential  to  the  solving  of  ill- 

126 

structured  problems  (King  and  Kitchener,  1994). 

Deanna  Kuhn  et  al.  provided  a  model  of  epistemology  that  links  epistemological  understanding 
to  critical  thinking  as  shown  in  Figure  3.  She  introduced  the  model  by  saying: 

Beliefs  about  knowing  and  knowledge  are  potentially  important  determinants  of 
intellectual  performance.  It  is  not  surprising  that  what  people  believe  about  the 
acquisition  of  knowledge  -  how  it  occurs  and  what  it  accomplishes  -  influences  its 
operation  in  their  own  lives.  Empirical  investigation  of  epistemological  beliefs,  however, 
has  not  been  widespread  and  has  received  relatively  little  attention  in  the  mainstream  of 
developmental  psychology,  despite  increasing  interest  on  the  part  of  developmental 

i  27 

psychologists  in  the  educational  implications  of  their  research. 


126  Hofer,  B.K.  "Personal  Epistemology  Research:  Implications  for  Learning  and  Teaching,"  Educational 
Psychology’  Review,  Vol.  13,  No.  4,  2001,  p.  365. 

127  Kuhn,  D.,  Cheney,  R.  &  Weinstock,  M.  "The  Development  of  Epistemological  Understanding,"  Cognitive 
Development,  Vol.  15,  No.  3,  2000,  pp.  309-310. 
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Levels  of  epistemological  understanding 


Level 

Assertions 

Reality 

Knowledge 

Critical  thinking 

Realist 

Assertions  arc  COPIES 
of  an  external  reality. 

Reality  is 
directly  knowable. 

Knowledge  conics 
from  an  external 
source  and  is  certain. 

Critical  thinking  is  unnecessary. 

Absolutist 

Assertions  are  FACTS 
that  arc  correct  or  incorrect 
in  their  representation  of  reality 
(possibility  of  false  belief). 

Reality  is 
directly  knowable. 

Knowledge  comes 
from  an  external 

source  and  is  certain. 

Critical  thinking  is  a  vehicle 
for  comparing  assertions  to 
reality  and  determining  their 
truth  or  falsehood. 

Multiplist 

Assertions  are  OPINIONS 
freely  chosen  by  and 
accountable  only  to  their  owners. 

Reality  is  not 
directly  knowable. 

Knowledge  is 
generated  by  human 
minds  and  is  uncertain. 

Critical  thinking  is  irrelevant. 

Fvaluativist 

Assertions  arc  JUDGMENTS 
that  can  be  evaluated  and 
compared  according  to  criteria 
of  argument  and  evidence. 

Reality  is  not 
directly  knowable. 

Knowledge  is 
generated  by  human 
minds  and  is  uncertain. 

Critical  thinking  is  valued  as  a 
vehicle  that  promotes  sound 
assertions  and  enhances  understanding. 

Figure  3:  Kuhn's  Levels  of  Epistemological  Understanding128 


Kuhn  explained  the  levels  of  epistemological  understanding  shown  in  Figure  3: 

We  propose  that  the  developmental  task  that  underlies  the  achievement  of  mature 
epistemological  understanding  is  the  coordination  of  the  subjective  and  objective 
dimensions  of  knowing.  Initially,  the  objective  dimension  dominates,  to  the  exclusion  of 
subjectivity.  Subsequently,  in  a  radical  shift,  the  subjective  dimension  assumes  an 
ascendant  position  and  the  objective  is  abandoned.  Finally,  the  two  are  coordinated,  with 
a  balance  achieved  in  which  neither  overpowers  the  other. 

This  progression  is  reflected  in  the  sequence  of  levels  depicted  in  Table  1.  Someone  at 
the  absolutist  (as  well  as  the  preabsolutist  realist)  level  sees  knowledge  as  an  objective 
entity,  as  located  in  the  external  world  and  knowable  with  certainty.  In  what  we  take  to  be 
a  key  event  in  the  development  of  epistemological  thought,  the  multiplist  relocates  the 
source  of  knowledge  from  the  known  object  to  the  knowing  subject,  hence  becoming 
aware  of  the  uncertain,  subjective  nature  of  knowing.  This  awareness  comes  to  assume 
such  proportions,  however,  that  it  overpowers  and  obliterates  any  objective  standard  that 
could  serve  as  a  basis  for  comparison  or  evaluation  of  conflicting  claims.  Because  claims 
are  subjective  opinions  freely  chosen  by  their  holders  and  everyone  has  a  right  to  their 
opinion,  all  opinions  are  equally  right. 

The  evaluativist  reintegrates  the  objective  dimension  of  knowing,  by  acknowledging 
uncertainty  without  forsaking  evaluation.  Thus,  two  people  can  both  have  legitimate 
positions  -  can  both  “be  right”  -  but  one  position  can  have  more  merit  (“be  more  right”) 

i  on 

than  the  other  to  the  extent  that  position  is  better  supported  by  argument  and  evidence. 

Personal  epistemology  has  not  yet  received  any  attention  in  the  intelligence  literature.  Because 
an  individual’s  personal  epistemology  affects  how  he  or  she  views  knowledge,  it  has  an  impact 
on  how  an  analyst  processes  information  and  how  willing  and  able  he  or  she  is  to  think  critically. 
It  is  a  concept  worthy  of  more  attention  in  the  intelligence  literature. 


12S  Kuhn,  p.  311. 

129  Kuhn,  pp.  310-312. 
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4.2.4.  Summary:  Enablers  of  Critical  Thinking 

I  developed  Table  10  as  a  model  of  critical  thinking  skills,  dispositions,  and  personal 
epistemology  that  may  be  useful  in  understanding  what  is  required  to  think  critically. 

Table  10:  Recommended  Critical  Thinking  Skills,  Dispositions,  and  Personal  Epistemology 


Critical  Thinking  Skills 

Critical  Thinking  Dispositions 

Interpretation 

Willingness  to  engage  in  intellectual 

Analysis 

effort 

Evaluation 

Systematic 

Inference 

Persistent 

Synthesis 

Intellectual  integrity 

Explanation 

Open-minded 

Self-regulation 

Intellectual  curiosity 

Personal  Epistemology 

Mature  level  that  enables  critical  thinking 

4.3  Can  We  Train  Someone  to  Think  Critically? 

We  now  understand  what  critical  thinking  is  and  what  skills,  dispositions,  and  personal 
epistemology  enable  someone  to  think  critically.  We  turn  our  attention  to  the  next  question:  Can 
we  train  someone  to  be  a  critical  thinker?  As  noted  earlier  in  the  report,  teaching  critical 
thinking  has  long  been  of  interest  as  demonstrated  by  the  United  States  Educational  Policies 
Commission  1938  report,  the  article  published  in  1939  in  the  New  York  Times  about  Cornell 
University’s  experiment  in  teaching  critical  thinking,  and  Edward  Glaser’s  1941  experiment  in 
teaching  critical  thinking. 

Despite  much  attention,  little  agreement  exists  about  how  to  teach  critical  thinking  effectively. 
Ongoing  questions  include  what  topics  to  address  when  teaching  critical  thinking;  what  teaching 
approaches  such  as  lecture,  discussion,  and  practice  are  most  effective;  how  much  time  it  takes 
for  someone  to  leam  to  think  critically  habitually  and  effectively;  and  whether  critical  thinking 
can  be  taught  as  an  independent  concept  or  can  only  be  taught  in  conjunction  with  a  specific 
topic  such  as  history. 

In  this  section  we  explore: 

•  Studies  about  and  approaches  for  infusing  critical  thinking  into  the  K-12  and 
college  educational  systems 

•  Lessons  learned  from  specific  experiments  and  longitudinal  studies 

•  Examples  of  critical  thinking  training  courses 

•  Summary  of  what  is  required  to  train  someone  to  think  critically 

4.3.1.  Infusing  Critical  Thinking  into  the  Educational  System 

Deanna  Kuhn  highlighted  the  fundamental  difficulty  of  teaching  thinking  to  K-12  students  in 
Education  for  Thinking :  what  content  should  be  included  when  teaching  critical  thinking?  This 
question  is  directly  related  to  the  widespread  inability  to  agree  on  what  critical  thinking  is  and 
what  skills  are  required  to  think  critically. 
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What  Are  Inquiry  Skills? 

Compare  a  teacher's  position  in  seeking  to  develop  students'  inquiry  skills  versus  their 
literacy  or  mathematical  skills.  In  the  latter  case,  the  teacher  can  draw  on  a  plethora  of 
finely  calibrated  assessment  instruments  that  have  been  widely  accepted  as  identifying  a 
student's  level  of  achievement  with  great  accuracy  -  a  student's  reading  level  is  routinely 
identified,  for  example,  as  year  six,  month  five.  Classroom  teachers  are  not  only  able  but 
in  fact  usually  required  to  undertake  frequent  assessments  of  these  skill  levels,  and 
evaluations  of  the  teacher's  own  performance  are  influenced  by  the  results.  There  is  little 
ambiguity  here  about  what  students  have  or  have  not  achieved.  Nor  is  there  any  scarcity 
of  curriculum  materials  designed  to  promote  this  achievement. 

Inquiry  skills,  in  contrast,  lack  even  a  clear  subject  area.  Most  often,  inquiry  skills  are 
regarded  as  part  of  science  education,  and  virtually  all  current  curriculum  standards  in 
science  education  include  inquiry  skills.  But  inquiry  skills  are  also  commonly  found  as 
curriculum  goals  in  Social  Studies  and  even  in  language  arts  (e.g.,  Levstik  and  Barton, 
2001,  p.  xi).  Y  et  even  in  science,  where  they  are  most  prevalent,  these  goals  and 
standards  remain  couched  in  the  most  general  of  terms.130 

Kuhn  continued: 

Even  the  most  conscientious  and  informed  teachers  are  left  with  little  to  go  on.  A  vast, 
seemingly  uncharted  terrain  remains  between  executing  the  activity  and  developing  the 
skills.  How  does  a  teacher  help  a  student  to  "analyze  and  interpret  data"  or  to  "think 
critically  and  logically  to  make  the  relationships  between  evidence  and  explanations"? 
What  teachers  have  to  settle  for  is  assessment  of  the  activities  themselves  rather  than  the 
skills  that  students  do  or  do  not  display  in  engaging  them.  If  an  activity  seems  a  "rich" 
one  -  that  is,  one  that  students  maintain  sustained  involvement  in  over  a  period  of  time 
without  losing  interest  -  then  it  is  deemed  an  inquiry  activity  successful  for  the  age  group. 
But  even  in  these  relatively  favorable  circumstances,  teachers  would  be  hard-pressed  to 

|ii 

identify  just  what  skills  the  students  are  acquiring  or  to  document  their  progress. 

Daniel  Willingham  noted  another  challenge  to  helping  K-12  students  learn  to  think  critically  - 
that  of  transfer.  Can  someone  take  the  knowledge  learned  in  the  classroom  and  apply  it  in 
multiple  situations? 

Since  the  ability  to  think  critically  is  a  primary  goal  of  education,  it’s  no  surprise  that 
people  have  tried  to  develop  programs  that  could  directly  teach  students  to  think  critically 
without  immersing  them  in  any  particular  academic  content.  But  the  evidence  shows  that 
such  programs  primarily  improve  students’  thinking  with  the  sort  of  problems  they 
practiced  in  the  program  -  not  with  other  types  of  problems.  More  generally,  it’s  doubtful 
that  a  program  that  effectively  teaches  students  to  think  critically  in  a  variety  of  situations 
will  ever  be  developed.  ' 

At  the  college  level,  Elizabeth  Jones  et  al.  noted  deficiencies  in  critical  thinking  education: 


130  Kuhn,  D.  Education  for  Thinking,  Harvard  University  Press,  Cambridge,  MA,  2005,  p.  40. 

131  Kuhn,  p.  41. 

132  Willingham,  D.T.  “Critical  Thinking:  Why  is  it  so  Hard  to  Teach  1 American  Educator, V ol.  31,  No.  2,  2007,  p. 
12. 
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Faculty  often  state  that  they  are  seeking  to  develop  students'  abilities  to  analyze, 
synthesize,  and  think  critically.  However,  research  indicates  that  faculty  do  not  follow 
their  good  intentions  when  they  develop  their  courses.  A  formal  review  and  analysis  of 
course  syllabi  and  exams  revealed  that  college  faculty  do  not  in  reality  focus  on  these 
advanced  skills  and  instead  are  far  more  concerned  with  students’  abilities  to  acquire 
knowledge,  comprehend  basic  concepts  or  ideas  and  terms,  and  apply  this  basic 
knowledge  (Ratcliff,  1994;  Ratcliff,  Jones,  Guthrie,  &  Oehler,  1991).  While  gathering 
data  on  actual  behaviors  is  more  difficult,  it  is  necessary  since  perceptions  do  not  match 
actions. 

Diane  Kelly-Riley  et  ah,  in  a  study  at  Washington  State  University,  confirmed  the  problems  with 
inconsistent  teaching  that  contributed  to  a  lack  of  learning  critical  thinking: 

As  we  expanded  our  pool  of  faculty  participants  in  the  Higher  Education  Coordinating 
Board  (HECB)  study,  we  found  that  some  instructors  demonstrated  a  substantial  need  for 
support  in  revising  their  practices  of  instruction  and  evaluation.  That  is,  their  habitual 
teaching  approaches  did  not  elicit  critical  thinking  from  their  students,  and  it  was  not  easy 
for  them  to  change  to  a  mode  that  would.134 

Despite  these  challenges,  there  has  been  tremendous  interest  in  the  topic  of  teaching  thinking.  A 
sample  of  the  books  and  articles  found  during  the  research  for  this  report  includes: 

135 

•  Thinking  and  Learning  to  Think  (1901) 

•  How  We  Think 136  (1910  andl933) 

•  “Teaching  to  Think  137  (193  3) 

•  “An  Experiment  in  Teaching  Certain  Skills  of  Critical  Thinking”138  (1944) 

•  Critical  Thinking  and  Education  (1981) 

•  “Needed:  Better  Methods  for  Teaching  Higher-Order  Thinking  Skills”140  (198  5) 

•  Teaching  Thinking:  Issues  and  Approaches1^  (1990) 

•  “Teaching  Thinking  Dispositions:  from  Transmission  to  Enculturation”142  (1993) 


133  Jones,  E.A.,  Hoffman,  S.,  Moore,  L.M.,  Ratcliff,  G.,  Tibbetts,  S.  &  Click,  B.A.L.  National  Assessment  of  College 
Student  Learning:  Identifying  College  Graduates'  Essential  Skills  in  Writing,  Speech  and  Listening,  and  Critical 
Thinking.  Final  Project  Report,  National  Center  on  Post  Secondary  Teaching,  Learning,  and  Assessment,  University 
Park,  PA,  1995,  p.  167. 

134  Kelly-Riley,  D.,  Brown,  G.,  Condon,  B.  &  Law,  R.  Washington  State  University  Critical  Thinking  Project,  The 
Center  for  Teaching,  Learning,  Technology,  General  Education  &  The  Writing  Programs,  Pullman,  WA,  2001,  p.  4. 

135  Schaeffer,  N.C.  Thinking  and  Learning  to  Think,  L.  B.  Lippincott  Co.,  Philadelphia  PA,  1901. 

136  Dewey,  J.  How  We  Think,  D.  C.  Heath  &  Co.,  Chicago,  IL,  1910  and  Dewey,  J.  How  We  Think:  A  Restatement 
of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed.,  Regnery,  Chicago,  IL,  1933. 

137  Kirby,  B.C.  "Teaching  to  Think,"  Journal  of  Education,  Vol.  116,  No.  14,  1933,  pp.  363-364. 

138  Anderson,  H.C.,  Marcham,  F.G.  &  Dunn,  S.B.  "An  Experiment  in  Teaching  Certain  Skills  of  Critical  Thinking," 
Journal  of  Educational  Research,  Vol.  38,  No.  4,  1944,  pp.  241-251. 

139  McPeck,  J.E.  Critical  Thinking  and  Education,  St.  Martin's  Press,  New  York,  NY,  1981. 

140  Quellmalz,  E.S.  "Needed:  Better  Methods  for  Teaching  Higher-Order  Thinking  Skills,"  Educational  Leadership, 
Vol.  43,  No.  2,  1985,  pp.  29-35. 

141  Swartz,  R.J.  &  Perkins,  D.N.  Teaching  Thinking:  Issues  and  Approaches,  Midwest  Publications,  Pacific  Grove, 
CA,  1990. 

143  Tishman,  S.,  Jay,  E.  &  Perkins,  D.N.  "Teaching  Thinking  Dispositions:  From  Transmission  to  Enculturation," 
Theory  into  Practice,  Vol.  32,  No.  3,  1993,  pp.  147-153. 
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An  interest  in  teaching  thinking  continues  to  the  present: 

•  “Learning  to  Think:  the  Challenges  of  Teaching  Thinking  143  (2005) 

•  Education  for  Thinking 144  (2005) 

The  majority  of  these  books  and  articles  proposed  theoretical  ways  to  teach  thinking  rather  than 
reporting  on  how  thinking  has  actually  been  taught.  Examples  of  real  world  practices  and  results 
are  included  later  in  this  report  in  the  “Lessons  Learned”  section. 

Numerous  critical  thinking  textbooks  are  available  for  use  at  the  college  level.  The  majority  of 
these  textbooks  focused  on  philosophy  and  logic  as  the  primary  vehicle  for  critical  thinking, 
resulting  in  a  less  comprehensive  perspective  on  critical  thinking  than  I  propose  in  this  report. 
Although  these  textbooks  include  useful  information,  none  are  sufficient  to  teach  the  level  and 
breadth  of  critical  thinking  needed  by  intelligence  analysts.  A  sample  of  titles  includes: 

•  Critical  Thinking:  An  Introduction  to  Logic  and  Scientific  Method 145  (1952, 
Philosophy/Logic) 

•  Critical  Thinking:  a  Functional  Approach 146  (1992,  Psychology) 

•  Thought  &  Knowledge:  An  Introduction  to  Critical  Thinking 147  (2003, 
Psychology) 

148 

•  Attacking  Faulty  Reasoning:  A  Practical  Guide  to  Fallacy-free  Arguments 
(2005,  philosophy/logic) 

•  Critical  Thinking 149  (2009,  Philosophy/Logic) 

•  Critical  Thinking  and  Communication 150  (2010,  Communications) 

A  number  of  organizations  are  dedicated  to  helping  infuse  critical  thinking  into  education.  They 
include: 

•  Foundation  for  Critical  Thinking  (www.criticalthinking.org) 

•  National  Center  for  Teaching  Thinking  (http://www.nctt.net/) 

•  Project  Zero  at  Harvard  University  (http://pzweb.harvard.edu/) 

•  The  Critical  Thinking  Company  (http://www.criticalthinking.com/index.isp) 

•  Association  for  Informal  Logic  and  Critical  Thinking  (http://ailact.mcmaster.ca/) 

•  Robert  Ennis’  resources  on  critical  thinking  (http://www.criticalthinking.net/) 


143  Ritchhart,  R.  &  Perkins,  D.N.  "Learning  to  Think:  The  Challenges  of  Teaching  Thinking,"  in  The  Cambridge 
Handbook  of  Thinking  and  Reasoning ,  eds.  K.  J.  Holyoak  &  R.G.  Morrison,  Cambridge  University  Press,  New  York, 
NY,  2005,  pp.  775-802. 

144  Kuhn,  D.  Education  for  Thinking,  Harvard  University  Press,  Cambridge,  MA,  2005. 

143  Black,  M.  Critical  Thinking:  An  Introduction  to  Logic  and  Scientific  Method,  2nd  ed.,  Prentice-Hall,  New  York, 
NY,  1952. 

146  Zechmeister,  E.B.  &  Johnson,  J.E.  Critical  Thinking:  A  Functional  Approach,  Brooks/Cole  Publishing  Co., 
Pacific  Grove,  CA,  1992. 

147  Halpern,  D.F.  Thought  and  Knowledge:  An  Introduction  to  Critical  Thinking,  Lawrence  Erlbaum  Associates, 
Mahwah,  NJ,  2003. 

148  Damer,  T.E.  Attacking  Faulty  Reasoning:  A  Practical  Guide  to  Fallacy-Free  Arguments,  5th  ed., 
Thomson/Wadsworth,  Belmont,  CA,  2005. 

149  Moore,  B.N.  &  Parker,  R.  Critical  Thinking,  9th  ed.,  McGraw-Hill,  Boston,  MA,  2009. 

150  Inch,  E.S.  &  Warnick,  B.  Critical  Thinking  and  Communication,  6th  ed.,  Allyn  &  Bacon,  Boston,  MA,  2010. 
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Despite  the  long-term  interest  in  infusing  critical  thinking  into  the  K-12  and  college  educational 
systems,  scholars  and  teachers  have  not  yet  agreed  upon  a  consistent,  widely  used  approach  for 
accomplishing  this  objective.  However,  I  did  find  isolated  examples  of  success  and  report  on 
these  examples  in  the  next  section  on  lessons  learned. 

4.3.2.  Lessons  Learned  About  Training  for  Critical  Thinking 

Numerous  publications  have  addressed  the  question  of  what  can  be  learned  from  previous 
attempts  to  teach  critical  thinking.  The  settings  that  generated  these  lessons  learned  fall  into  one 
of  three  categories: 

•  General  studies  such  as  evaluations  of  how  successful  college  critical  thinking 
classes  are  in  teaching  transferable  skills  or  studies  of  the  impact  of  college  on 
critical  thinking  skills 

•  Specific  and  controlled  critical  thinking  training  experiments  such  as  done  by 
Edward  Glaser,  Jennifer  Reed,  and  several  universities 

•  Findings  from  years  of  teaching  critical  thinking  in  the  nursing  profession 

Results  of  General  Studies:  General  studies  report  mixed  results,  with  some  describing 
successful  outcomes  and  others  reporting  a  lack  of  success.  Several  studies  highlighted  the 
challenge  of  teaching  something  and  testing  for  it  when  the  definition  of  the  concept  varies  and 
when  no  agreed-upon  instructional  best  practices  or  assessment  mechanisms  exist.  Considering 
the  conflicting  definitions  of  critical  thinking,  the  variety  of  skills  and  dispositions  identified  as 
supporting  critical  thinking,  and  the  inconsistency  in  instructional  approaches,  the  mixed  results 
reported  in  the  studies  are  not  surprising. 

Stephen  Norris  reported  little  evidence  to  support  the  effectiveness  of  critical  thinking  instruction 
in  his  1985  research  synthesis.  He  also  highlighted  the  challenge  of  assessing  the  degree  to 
which  someone  has  learned  critical  thinking. 

There  Is  Little  Detailed  Knowledge  About  the  Effectiveness  of  Teaching  Critical 
Thinking. 

Research  on  the  effectiveness  of  critical  thinking  instruction  almost  invariably  uses 
indicators  of  effectiveness  that  are  insensitive  to  fine  details.  The  research  typically 
concludes  that  instruction  is  effective  (Annis  and  Annis,  1979;  Frank,  1969;  Moll  and 
Allen,  1982;  Ross  and  Semb,  1981;  Wolf  and  others,  1968;  Wright,’  1977;  Yeazell, 

1981).  In  these  studies,  and  ones  like  them,  classes  of  students  experience  a  treatment 
designed  to  improve  some  aspect  of  their  thinking  ability.  The  treatments  usually  consist 
of  a  unit  of  work  extending  over  a  few  weeks  or  as  long  as  a  year,  and  are  based  on  the 
intuitions  of  the  researchers  about  what  ought  to  be  effective  instruction  in  thinking.  The 
criterion  for  determining  whether  or  not  the  treatment  has  a  positive  effect  is  often  one  of 
the  general  critical  thinking  tests  mentioned  earlier,  or  a  test  designed  specifically  for  the 
study. 

Many  of  the  studies  do  not  use  control  groups,  so  special  care  must  be  taken  when 
interpreting  their  results.  Regardless  of  any  specific  limitations  on  the  research  design, 
however,  two  issues  emerge.  There  is  little,  if  any,  evidence  on  the  long-term  impact  of 
instruction  in  critical  thinking,  despite  the  fact  that  the  vision  of  such  impact  is  central  to 
the  justification  of  critical  thinking  instruction.  In  addition,  while  the  conclusion  of  the 
studies  is  usually  that  instruction  leads  to  better  critical  thinkers,  we  do  not  leam  what 
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specifically  makes  these  students  better  thinkers  and  in  what  specific  ways  they  can  still 
improve.  Are  they  better  thinkers  because  they  have  acquired  a  greater  knowledge  of 
principles  of  thinking,  such  as  those  proposed  by  Ennis,  or  because  they  tend  to  monitor 
more  skillfully  the  progress  of  their  own  thinking,  or  because  they  have  more  completely 
adopted  Siegel's  critical  spirit?151 

Diane  Halpern  drew  a  different  conclusion  from  a  2001  study: 

The  Evidence 

Despite  all  of  the  difficulties  in  assessing  gains  in  critical  thinking,  there  is  a  diverse  body 
of  evidence  showing  that  thinking  can  be  improved  with  instruction  that  is  specifically 
designed  for  that  purpose.  Although  there  has  been  some  debate  about  whether  it  is 
possible  to  produce  long  lasting  enhancement  in  the  ability  to  think  effectively  (Block, 
1985;  Glaser,  1984;  Resnick,  1983),  we  now  have  a  considerable  body  of  evidence  that 
thinking-skills  courses  have  positive  effects  that  are  transferable  to  a  wide  variety  of 
situations.  At  least  seven  qualitatively  different  forms  of  outcome  evaluations  for 
thinking  courses  have  been  conducted,  all  of  which  are  generally  supportive  of  the  idea 
that  the  ability  to  think  critically  can  be  improved  as  a  result  of  thinking-skills  instruction. 

While  none  of  these  constitute  the  “perfect  measure,”  taken  together  they  provide  a 
substantial  body  of  support  for  the  value  of  thinking-skills  instruction.152 

Ernest  T.  Pascarella  and  Patrick  T.  Terenzini  have  completed  a  series  of  evaluations  of  the 
impact  college  has  on  students.  One  topic  they  address  is  how  the  overall  college  experience 
fosters  critical  thinking.  They  compared  findings  over  time;  Table  1 1  summarizes  their 
results.  Note  that  these  results  come  from  different  studies  using  different  assessment 
methodologies. 

Table  11:  Critical  Thinking  Improvement  of  Upper  Class  Students  Compared  to 

Freshmen 


Year 

Critical  Thinking  Improvement 

1991 

1  standard  deviation  improvement  of  seniors  to  freshmen, 
giving  a  34  percentile  point  advantage 

1997 

.54  standard  deviation  improvement  of  seniors  to  freshmen, 
giving  a  2 1  percent  advantage 

1999 

.37  standard  deviation  improvement  of  juniors  to  freshmen, 
giving  a  14  percentile  point  advantage 

Pascarella  and  Terenzini  summarized  their  findings: 

Although  change  during  college  is  quite  often  an  inaccurate  estimate  of  the  actual  impact 
of  college,  the  evidence  from  the  1990s  nevertheless  suggests  that  students  are  making 
gains  in  critical  thinking  skills  during  college  that  are  appreciably  smaller  in  magnitude 
than  the  gains  we  observed  in  our  previous  synthesis.  It  is  not  readily  apparent  from  the 


151  Norris,  S.P.  "Syntheses  of  Research  on  Critical  Thinking,"  Educational  Leadership,  Vol.  42,  No.  8,  1985,  pp.  44- 
45. 

152  Halpern,  D.F.  "Assessing  the  Effectiveness  of  Critical  Thinking  Instruction,"  Journal  of  General  Education,  Vol. 
50,  No.  4,  2001,  p.277. 

153  Pascarella,  E.T.  &  Terenzini,  P.T.  How  College  Affects  Students:  A  Third  Decade  of  Research,  Jossey-Bass,  San 
Francisco,  CA,  2005,  pp.  156-158. 

54 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 

88ABW-2014-1598,  Cleared  11  April  2014 


evidence  why  this  is  the  case.  However,  the  more  modest  evidence  of  change  in  the 
1990s  does  underscore  an  important  point.  Not  all  students  develop  as  critical  thinkers 
during  college.154 

Claudia  Maria  Alvarez  Ortiz  completed  a  meta-analysis  of  studies  concerning  the  effectiveness 
of  various  approaches  to  teaching  critical  thinking.  The  primary  question  in  her  study  was 
whether  or  not  the  study  of  philosophy  improves  critical  thinking  abilities.  Her  findings  include: 

This  meta-analysis  gives  us  the  best  yet  fix  on  the  extent  to  which  critical  thinking  skills 
improve  over  a  semester  of  studying  philosophy,  general  university  study,  and  studying 
critical  thinking.  The  meta-analysis  results  indicate  that  students  do  improve  while 
studying  philosophy,  and  apparently  more  so  than  general  university  students,  though  we 
cannot  be  very  confident  that  this  difference  is  not  just  the  result  of  random  variation. 
More  importantly,  studying  philosophy  is  less  effective  than  studying  critical  thinking, 
regardless  of  whether  one  is  being  taught  in  a  philosophy  department  or  in  some  other 
department.  Finally,  studying  philosophy  is  much  less  effective  than  studying  critical 
thinking  using  techniques  known  to  be  particularly  effective  such  as  LAMP.155 

Richard  Arum  and  Josipa  Roksa  conducted  a  multi-year,  multi-institution  study  of  college 
learning  and  published  the  results  in  201 1.  Arum  and  Roksa  noted  the  importance  US  higher 
education  places  on  critical  thinking  while  pointing  out  that  the  reality  does  not  match  the 
rhetoric.  This  concurs  with  what  Jones  and  Kelly-Riley  stated,  as  reported  earlier  in  this 
document. 

Indeed,  99  percent  of  college  faculty  say  that  developing  students’  ability  to  think 
critically  is  a  “very  important”  or  “essential”  goal  of  undergraduate  education.  87%  also 
claim  that  promoting  students’  ability  to  write  effectively  is  “very  important”  or 
“essential.”12 

However,  commitment  to  these  skills  appears  more  a  matter  of  principle  than  practice,  as 

the  subsequent  chapters  in  this  book  document.  The  end  result  is  that  many  students  are 
only  minimally  improving  their  skills  in  critical  thinking,  complex  reasoning,  and  writing 
during  their  journeys  through  higher  education.  From  their  freshman  entrance  to  the  end 
of  their  sophomore  year,  students  in  our  sample  on  average  have  improved  these  skills,  as 
measured  by  the  CLA,  by  only  0. 18  standard  deviation.13  This  translates  into  a  seven 
percentile  point  gain,  meaning  that  an  average-scoring  student  in  the  fall  of  2005  would 
score  seven  percentile  points  higher  in  the  spring  of  2007. 156 

Arum  and  Roksa  referred  to  the  Pascarella  and  Terenzini  studies  of  college  learning  and  compare 
current  learning  with  the  amount  students  achieved  in  the  past: 

How  do  we  know  that  a  0. 18  standard  deviation  does  not  represent  remarkable  growth? 

. . .  Summarising  an  extensive  body  of  research,  Pascarella  and  Terenzini  estimated  that 
seniors  had  a  0.50  standard  deviation  advantage  over  freshmen  in  the  1990s.  In  contrast, 


154  Pascarella,  p.  158. 

155  Ortiz,  C.M.A.  Does  Philosophy  Improve  Critical  Thinking  Skills?  MA  thesis,  University  of  Melbourne,  2007,  pp. 
ii-iii.  Lots  of  Argument  Mapping  Practice  (LAMP),  refers  to  classes  that  include  extensive  practice  rather  than 
traditional  lecture  and  testing.  An  example  of  critical  thinking  training  with  LAMP  is  described  in  the  next  section  in 
the  experiment  conducted  by  Tim  van  Gelder. 

156  Arum,  R.  &  Roksa,  J.  Academically  Adrift:  Limited  Learning  on  College  Campuses,  University  of  Chicago  Press, 
Chicago,  IL,  2011,  p.  35. 
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during  the  1980s  students  developed  their  skills  at  twice  the  rate:  seniors  had  an 
advantage  over  freshman  of  one  standard  deviation.157 

The  general  studies  of  growth  in  critical  thinking  identify  two  key  findings: 

•  Non-specific  interventions  such  as  simply  attending  college  do  not  result  in  strong 
growth  of  critical  thinking  abilities. 

•  The  more  specific  the  intervention,  the  stronger  the  outcome.  As  Ortiz  reported, 
critical  thinking  classes  resulted  in  stronger  abilities  than  logic  classes,  and  critical 
thinking  classes  with  extensive  practice  had  the  strongest  outcome  of  all. 

The  experiments  described  in  the  next  section  identify  approaches  that  do  successfully  increase 
critical  thinking  abilities. 

Critical  Thinking  Training  Experiments:  Several  researchers  wrote  about  specific  experiments 
in  teaching  critical  thinking,  and  all  reported  successful  outcomes.  The  common  success  factors 
in  these  experiments  include: 

1 .  Identify  a  clear  desired  outcome  -  what  is  the  improvement  in  critical  thinking  expected 
to  accomplish? 

2.  Develop  an  understandable  description  of  critical  thinking  that  addresses  the  desired 
outcome  and  clearly  describes  needed  competencies. 

3.  Design  the  program  to  specifically  develop  critical  thinking  competencies  in  the  critical 
thinking  description  and  that  will  enable  the  students  to  achieve  the  desired  outcome. 

a.  Topics,  contents,  and  materials 

b.  Teaching  methodologies 

c.  Opportunities  for  meaningful  practice 

d.  Faculty  well  trained  in  all  of  the  above 

e.  Consistent  implementation  by  all  faculty  involved 

4.  Secure  or  develop  an  assessment  that  confirms  that  students  learned  what  was  intended 
and  achieved  the  desired  outcome. 

Edward  Glaser’s  An  Experiment  in  the  Development  of  Critical  Thinking  described  the  results  of 
his  work  with  school  children.  He  defined  his  intended  outcome,  developed  specific  teaching 
materials  and  practices  designed  to  achieve  this  outcome,  conducted  the  experiment,  and 
compared  findings  with  those  from  a  control  group. 

The  chief  problems  of  this  study  are: 

1.  To  develop  and  present  materials  and  illustrative  teaching  procedures  which  may  be 
used  effectively  by  the  teacher  of  upper-grade  elementary,  secondary,  and  college 
students  to  cultivate  a  spirit  of  inquiry  and  to  stimulate  growth  in  ability  to  think 
critically. 

2.  To  evaluate  the  effectiveness  of  these  materials  and  teaching  procedures. 

3.  To  ascertain  whether  or  not  there  is  a  relationship  between  ability  to  think  critically 
and  certain  other  factors  such  as  intelligence,  reading  ability,  patterns  of  interest-values, 
home  background,  and  sex,  and  to  determine  whether  these  factors  are  significantly 


Arum ,  pp.  35-36. 
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associated  with  amount  of  gain  on  the  critical  thinking  tests  after  receiving  special 

•  •1  SR 

instruction  in  critical  thinking. 

The  instruction  consisted  of  eight  topics  supportive  of  critical  thinking: 

There  were  eight  lesson  units  in  the  original  series  developed  for  use  by  the  teachers  of 
the  experimental  classes.  They  were  entitled: 

(1)  Recognition  of  Need  for  Definition 

(2)  Logic  and  the  Weight  of  Evidence 

(3)  The  Nature  of  Probable  Inference 

(4)  Deductive  and  Inductive  Inference 

(5)  Logic  and  the  Method  of  Science,  and  Some  Characteristics  of  Scientific  Attitude 

(6)  Prejudice  as  a  Factor  Making  for  "Crooked  Thinking," 

(7)  Values  and  Logic 

(8)  Propaganda  and  "Crooked  Thinking."159 

Each  unit  included  time  in  the  classroom  with  lecture  and  practice  as  well  as  homework  intended 
to  enable  the  students  to  connect  the  concepts  to  daily  life.  Each  unit  also  included  “knowledge 
objectives”  and  “appreciation  objectives”  which  we  now  call  dispositions. 

Glaser  reported  the  following  positive  results: 

Conclusions: 

1 .  The  average  gain  (in  composite  Z-score)  on  the  battery  of  critical  thinking  tests  of  the 
four  experimental  classes  after  ten  weeks  was  significantly  greater  than  the  average  gain 
of  the  four  control  classes  on  the  same  tests  (diff./oa=6.09).  This  significant  difference 
supports  the  conclusion  that  the  lesson  units  which  have  been  developed  can  be  used 
effectively  with  high  school  (and  probably  college)  students  to  stimulate  growth  in  ability 
to  think  critically. 

2.  The  improvement  in  ability  to  think  critically  appears  to  be  somewhat  general  in 
character.  There  is  evidence  for  some  students  of  improvement  in  general  disposition  to 
consider  problems  thoughtfully,  and  evidence  of  ability  to  think  more  critically  in  other 
classes  in  school,  at  home,  in  connection  with  personal  problems,  and  in  connection  with 
speeches,  advertisements,  and  arguments.  The  aspect  of  critical  thinking  which  appears 
most  susceptible  to  general  improvement  is  the  attitude  of  being  disposed  to  consider  in  a 
thoughtful  way  the  problems  and  subjects  that  come  within  the  range  of  one's  experience. 
An  attitude  of  wanting  evidence  for  beliefs  is  most  subject  to  general  transfer.160 

Donald  Hatcher  of  Baker  University  described  the  approach  an  entire  university  took  to  improve 
critical  thinking  in  its  undergraduate  students  in  “Critical  Thinking  and  the  Liberal  Arts.” 

Hatcher  wrote  a  sequel  that  reported  the  results  of  the  experiment  in  “Stand-alone  Versus 
Integrated  Critical  Thinking  Courses.”  Baker  University  established  a  cross-disciplinary  team  to 
develop  a  course  required  of  all  freshmen  in  order  to  improve  their  critical  thinking  and  writing 
skills.  The  genesis  of  the  project  was  a  realization  that  students  were  ill-prepared  to  produce  a 


158  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  p.8. 

159  Glaser,  p.  94. 

160  Glaser,  p.  175. 
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senior  capstone  project,  which  required  students  to  write  a  15-20  page  paper  arguing  for  a 
specific  public  policy  on  a  science,  technology,  or  human  values  issue: 

The  senior  capstone  began  1979.  It  was  not  long  before  those  teaching  the  course  realized 
that  many  Baker  University  seniors  were  seriously  challenged,  to  put  it  mildly,  when 
asked  to  write  such  a  paper.  The  primary  difficulty  was  that  students  did  not  understand 
how  to  construct  or  evaluate  arguments.  In  1988,  a  group  of  professors  from  the 
humanities  began  planning  the  freshman  sequence  with  the  goal  of  addressing  this 
shortcoming.161 

Hatcher  served  as  project  leader.  He  required  the  full  collaboration  and  participation  of  other 
teachers  in  developing  and  delivering  the  program.  This  shared  responsibility  created  challenges 
which  took  effort  to  overcome: 

In  our  attempts  to  create  a  new  sequence,  the  main  activities  involved  getting  clear  on 
what  we  wanted  the  course  to  accomplish  (instruction  in  critical  thinking  and  writing 
coupled  with  learning  to  read  the  classic  texts).  Once  we  were  clear  on  this,  we 

developed  our  own  text  to  achieve  our  goals.  The  process  of  ten  faculty  writing  a  text 

1 6 2 

was  a  trying,  but  ultimately  fulfilling  experience. 

Hatcher  emphasized  two  important  success  factors:  gaining  agreement  on  the  concept  of  critical 
thinking  to  be  taught  and  verifying  that  the  class  does  achieve  the  needed  outcomes. 

First,  it  is  never  easy  to  achieve  agreement  in  a  group  of  faculty  from  different 
disciplines.  Enthusiasm  is  not  the  same  as  “likemindedness.”  Anyone  who  desires  to  be  a 
project  director  should  be  prepared  for  confrontational  situations.  Second,  from  the 
beginning  each  participant  must  be  willing  to  compromise  or  change  a  position  in  light  of 
new  arguments.  Third,  because  projects  seldom  progress  on  schedule,  it  is  better  in 
program  development  to  plan  for  at  least  two  years  with  ample  time  for  theoretical 
discussions,  trial  runs,  and  revisions.  Only  through  running  trial  semesters  for  three 

1  ZTT 

semesters  were  we  able  to  determine  what  approaches  and  reading  worked  best. 

Hatcher’s  analysis  of  what  was  accomplished  after  sixteen  years  of  experience  is  equally 
interesting.  He  emphasized  the  importance  of  a  common  understanding  of  critical  thinking.164 

For  those  who  worked  on  the  project,  getting  clear  on  exactly  what  is  meant  by  critical 
thinking  was  the  first  task.  As  previously  noted,  one’s  specific  conception  of  critical 
thinking  will  influence  greatly  both  the  structure  and  the  content  of  the  courses.  The 
group  examined  some  of  the  standard  definitions  and  was  not  completely  comfortable 
with  any  of  them  (Hatcher,  2000).  It  was  important  that  the  definition  be  as  clear  and 
concise  as  possible,  so  that  both  those  planning  the  courses  and  the  unconvinced  among 
the  faculty  would  know  what  was  being  discussed  when  the  new  sequence  was  proposed 
as  a  part  of  Baker’s  General  Education  Program  and  required  of  all  freshmen.  The 
definition  should  make  it  easy  to  see  why  critical  thinking  is  an  important  educational 
goal.  It  should  also  refer  specifically  to  the  criteria  to  be  used  for  critical  judgment. 
Otherwise,  there  will  never  be  agreement  over  what  counts  as  a  “reasonable  position.”  It 


161  Hatcher,  D.L.  "Stand-Alone  Versus  Integrated  Critical  Thinking  Courses,"  Journal  of  General  Education,  Vol. 
55,  No.  3-4,  2006,  p.250. 

162  Hatcher,  D.L.  Critical  Thinking  and  the  Liberal  Arts,  Baker  University,  Baldwin  City,  KS,  1990,  pp.  5-6. 

163  Hatcher,  p.  6. 

164  The  Baker  University  definition  of  critical  thinking  was  included  in  section  4. 1.2. 2  of  this  report. 
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should  allow  people  to  distinguish  critical  thinking  from  other  cognitive  activities  such  as 
creative  thinking,  problem  solving,  and  logical  inference.165 

Baker  University  took  a  unique  approach  to  developing  the  content  of  the  program,  choosing  to 
include  minimal  theory  with  a  greater  emphasis  on  applying  the  theory  through  extensive 
practice  during  a  two-course  sequence.  This  approach  supports  what  Ortiz  found  in  her  meta¬ 
analysis. 

Only  three  to  four  weeks  are  spent,  an  unusually  brief  time  compared  with  even  one- 
semester  critical  thinking  courses,  studying  deduction,  induction,  and  a  few  of  the  more 
common  informal  fallacies.  The  final  weeks  of  the  semester  provide  instruction  in  how 
these  logical  tools  can  be  applied  when  writing  papers.  With  minimal  instruction, 
students  can  use  some  of  the  standard  argument  patterns,  such  as  modus  ponens,  modus 
tollens,  disjunctive  syllogism,  or  some  combination  of  these,  to  construct  and  evaluate 
arguments  in  support  of  positions  they  might  defend  in  a  paper.166 

These  two  courses  differ  from  traditional  critical  thinking  courses  insofar  as  they  focus 
on  the  use  of  formal  logic  and  CT  skills  to  argue  for  and  critique  positions  in  student 
papers.  Throughout  the  two  semesters,  the  time  spent  on  writing,  approximately  80%,  far 
exceeds  the  time  spent  on  instruction  in  the  logical  tools  necessary  to  evaluate  ideas 
critically.  They  differ  from  traditional  written  composition  courses  insofar  as  they 
emphasize  only  one  type  of  paper —  the  argumentative  essay.167 

Although  the  articles  imply  that  performance  on  the  senior  capstone  project  improved,  the 
reports  from  Hatcher  did  not  explicitly  identify  or  quantify  these  improvements.  Rather,  the 
reports  identified  improvements  as  measured  by  standardized  critical  thinking  assessments. 

Baker  University  used  a  variety  of  assessments  to  determine  whether  the  students  did  further 
develop  their  critical  thinking  skills.  All  assessments  over  15  years’  experience  indicated 
significant  improvement  in  critical  thinking  skills.  The  results  of  the  study  exceeded  what  could 
be  expected  from  a  typical  college  experience.  The  results  from  Baker  University  are  more 
significant  than  those  reported  by  Pascarella  &  Terenzini  and  by  Arum  &  Roksa. 

As  Table  1  indicates,  the  average  effect-size  gains  by  the  Baker  students  are  significantly 
higher  (+1.47). 168 

Jennifer  Reed  conducted  an  experiment  to  teach  critical  thinking  as  part  of  history  classes.  As 
with  Glaser,  Reed  had  a  clear  understanding  of  what  outcomes  she  wanted  to  achieve  and  how 
she  planned  to  achieve  them. 

Statement  of  the  Problem: 

This  study  investigated  the  effect  of  integrating  Richard  Paul’s  model  for  critical  thinking 
into  a  U.S.  history  course  on  community  college  students’  1)  abilities  to  think  critically 
about  U.S.  history  and  about  everyday  issues,  2)  dispositions  toward  thinking  critically, 


165  Hatcher,  D.L.  "Stand-Alone  Versus  Integrated  Critical  Thinking  Courses,"  Journal  of  General  Education,  Vol. 
55,  No.  3-4,  2006,  pp.  250-251. 

166  Hatcher,  p.  253. 

167  Hatcher, ,  p.  254. 

168  Hatcher,  p.  255. 
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and  3)  knowledge  of  history  content.  This  study  also  examined  if  age  (under  22,  22  and 
older)  or  gender  moderated  the  effectiveness  of  the  instructional  method.169 

The  major  findings  of  this  study  can  be  summarized  as  follows: 

Community  college  students’  abilities  to  think  historically  and  to  think  critically  can 
improve  in  a  single  course  when  provided  with  explicit  and  intensive  training. 

Community  college  students’  end  of  term  knowledge  of  history  content  need  not  suffer 
when  explicit  training  in  critical  thinking  abilities  has  been  integrated  into  course 
material.170 

Reed  also  noted  that  her  experiment  had  no  impact  on  thinking  dispositions. 

To  test  students’  dispositions  toward  critical  thinking,  students  in  both  groups  took  the 
CCTDI  during  the  first  two  weeks  of  the  course  and  again  during  the  last  week  of  the 
course.  Results  from  statistical  analyses  of  the  scores  on  this  instrument  showed  no 
significant  differences  between  the  experimental  and  control  groups.  Further,  posttest 
means  were  not  significantly  different  from  pretest  means  in  either  group  (Pretest: 
Experimental  M  =  296,  Control  M  =  297;  Posttest:  Experimental  M  =  298,  Control  M  = 
302).  It  appears  that  taking  a  single  history  course  that  includes  explicit  instruction  in 
Paul’s  model  but  does  not  emphasize  intellectual  traits  of  the  critical  thinker  has  no  effect 
on  students’  dispositions  toward  critical  thinking.171 

Tim  van  Gelder  wrote  about  improvements  in  results  from  a  college  critical  thinking  class  when 
automated  argument  mapping  software  is  part  of  the  program.  Van  Gelder  tested  the  theory  of 
quality  practice  using  a  software  tool  called  ReasonlAble.  ReasonlAble  assists  with  developing 
and  evaluating  arguments,  which  provides  practice  in  interpretation,  analysis,  evaluation,  and 
inference  skills. 

Reviewing  the  research  literature  on  cognitive  skill  acquisition,  we  found  a  consensus 
around  the  unsurprising  idea  that  cognitive  skills,  like  other  skills,  improve  with  practice. 
However,  it  must  be  the  right  kind  of  practice.  Generically,  the  practice  should  be 

•  Motivated  -  the  student  should  be  deliberately  practicing  in  order  to  improve  skills 

•  Guided  -  the  student  should  have  some  way  of  knowing  what  to  do  next 

•  Scaffolded  -  particularly  in  early  stages,  there  should  be  structures  preventing 
inappropriate  activity 

•  Graduated  -  tasks  should  gradually  increase  in  complexity 

•  Feedback  -  the  student  should  have  some  way  of  telling  whether  a  particular 

1  77 

activity  was  successful  or  appropriate 

From  the  outset  we  have  been  aiming  to  develop  an  approach  to  critical  thinking 
instruction  that  demonstrably  produces  substantial  gains.  (A  further  constraint  is  that  the 


lh9  Reed,  J.H.  Effect  of  a  Model  for  Critical  Thinking  on  Student  Achievement  in  Primary  Source  Document 
Analysis  and  Interpretation,  Argumentative  Reasoning,  Critical  Thinking  Dispositions,  and  History  Content  in  a 
Community  College  History  Course,  Ph.  D.  dissertation.  University  of  South  Florida,  1998,  p.  xi. 

170  Reed,  p.  160. 

171  Reed,  p.  150. 

172  van  Gelder,  T.  "How  to  Improve  Critical  Thinking  Using  Educational  Technology,"  Meeting  at  the  Crossroads: 
Proceedings  of  the  Annual  Conference  of  the  Australasian  Society  for  Computers  in  Learning  in  Tertiary >  Education 
(ASCILITE  2001),  Melbourne,  Australia,  9-12  December  2001,  p.  540. 
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approach  be  practical  and  affordable  for  widespread  use.)  The  new  approach  is  the 
Reason!  method,  which  essentially  just  implements  the  QPH  with  Reason!  Able  as  the 

•  •  173 

practice  environment. 

Results: 

Data  is  available  for  studies  conducted  in  both  2000  and  2001.  The  two  studies  yielded 
essentially  the  same  results.  The  CCTST  indicated  improvement  with  an  effect  size  of 
about  0.84  (mean  difference  between  paired  pre-  and  post-test  scores,  divided  by  standard 
deviation  on  the  pre-test).  The  written  test  showed  effect  sizes  of  a  similar  order  of 
magnitude,  though  there  were  substantial  differences  and  low  correlations  between 
scorers.  Collectively  these  results  indicate  that  the  effect  is  quite  robust.  Crudely 
speaking,  we  are  seeing  a  gain  in  critical  thinking  of  the  best  part  of  one  SD  in  a  single 
semester.  (For  comparison,  an  equivalent  shift  in  IQ  would  be  about  12-13  points  on 
average  across  the  group.)17 

All  of  these  examples  demonstrate  that  a  well-designed  approach  to  teaching  critical  thinking  can 
improve  students’  critical  thinking  performance. 

Findings  from  Years  of  Teaching  Critical  Thinking  in  Nursing  Programs:  The  nursing 
profession  embraced  critical  thinking  as  a  key  skill  for  nurses.  As  a  result,  education  for  critical 
thinking  in  nursing  has  been  extensively  studied.  Numerous  articles  about  critical  thinking  as 
applied  to  nursing  described  what  has  been  learned  from  over  a  decade  of  emphasis  on  critical 
thinking  as  an  essential  outcome  of  nursing  education. 

It  is  not  clear  that  this  emphasis  on  critical  thinking  has  had  a  significant  impact  on  measurable 
critical  thinking  ability.  Some  of  the  reasons  for  this  lack  of  results  include: 

•  Lack  of  a  nursing-specific  definition  and  conception  of  critical  thinking 

•  Lack  of  a  consistent  understanding  of  how  to  apply  critical  thinking  concepts  to 
nursing 

•  Lack  of  effective,  consistent  teaching  approaches  and  methods. 

•  Multiple  assessment  methods  that  do  not  demonstrate  what  has  been  learned 
effectively  and  that  are  not  specific  to  nursing 

A  1998  study  of  critical  thinking  in  nursing  education  noted: 

In  nursing  education,  critical  thinking  is  not  a  clearly  understood  concept,  nor  is  it 
systematically  applied.175 

Sharon  Staib  conducted  an  analysis  of  studies  about  teaching  and  assessing  critical  thinking  in 
nursing  in  an  attempt  to  identify  effective  teaching  strategies.  The  article  included  mention  of  a 
Delphi  process  conducted  in  2000  to  define  critical  thinking  specifically  for  nursing.  This  effort 
resulted  in  the  following  definition: 

Critical  thinking  is  an  essential  component  of  professional  accountability  and  quality 
nursing  care.  Critical  thinkers  in  nursing  exhibit  these  habits  of  the  mind:  confidence, 
contextual  perspective,  creativity,  flexibility,  inquisitiveness,  intellectual  integrity, 


van  Gelder,  p.  545. 

174  van  Gelder,  p.  546. 

175  Myrick,  F.  Preceptorship  and  Critical  Thinking  in  Nursing  Education,  Ph.D.  dissertation.  University  of  Alberta, 
1998,  p.  23. 
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intuition,  open-mindedness,  perseverance,  and  reflection.  Critical  thinkers  in  nursing 
practice  the  cognitive  skills  of  analyzing,  applying  standards,  discriminating,  information 
seeking,  logical  reasoning,  predicting  and  transforming  knowledge.176 

Staib’s  study177  demonstrated  the  danger  of  calling  for  the  inclusion  of  a  topic  into  a  national 
curriculum  without  adequate  advance  preparation.  Staib  compiled  a  table,  shown  in  Figure  4, 
summarizing  studies  of  critical  thinking  teaching  strategies,  goals,  assessments,  and  results.  She 
found: 

•  Wide  variety  of  teaching  strategies  (journaling,  students  interviewing  mothers  about 
their  birth  experience,  case  studies,  concept  maps,  and  more) 

•  Limited  content  addressed  in  the  teaching.  Of  the  ten  habits  identified  in  the 
consensus  definition  above,  none  of  the  training  programs  addressed  all  of  the 
habits.  Two  programs  addressed  four  of  the  habits,  while  the  remaining  programs 
addressed  between  zero  and  three. 

•  Multiple  assessment  methods  (fonnal  CT  assessments,  anecdotal/self  report) 


176  Norris,  S.P.  &  Ennis,  R.H.  Evaluating  Critical  Thinking,  Midwest  Publications,  Pacific  Grove,  CA,  1989,  p.  499. 

177  Staib,  S.  “Teaching  and  Measuring  Critical  Thinking,”  Journal  of  Nursing  Education,  Vol.  42,  No.  11,  2003,  pp. 
500-501. 
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TABLE 

Summary  of  Studies  Using  Critical  Thinking  Strategies 

Author 

Method 

Goal 

Evaluation 

Method 

Critical  Thinking 
“Habits  of  the  Mind" 

Results  Addressed 

Baker  (1996) 

Journaling 

To  help  students  identify 
their  own  cognitive  and 
affective  behavior  during 
nursing  practice 

Reflection 

Bell  et  at 
(2002) 

Six-step, 
process-focused 
group  learning 
strategy 

To  teach  students  how  to 
conduct  in-depth  analyses  of 
clinical  incidents  that  caused 
them  to  use  critical  thinking 

Anecdotal 

Students  appeared  to 
strengthen  theoretical  and 
experiential  knowledge. 

Reflection, 

creativity, 

open-mindedness 

Callister  (1996) 

Stu  dents 
interviewed  their 
own  mothers 
about  their  birth 
experience 

To  foster  critical  thinking  in 
the  cognitive  and  affective 
domains 

Anecdotal 

Student  and  faculty 
reported  an  increase  in 
critical  thinking. 

Reflection, 

contextual 

perspective 

Daley  et  at 
(1999) 

Concept  maps 

To  increase  critical  thinking 
by  helping  students  identify 
and  draw  relationships 
between  concepts 

•  Evaluated 
knowledge  acquired 
during  the  semester 

•  Statistical 
comparison  of 
first  concept  map 

to  third  concept  map 

•  Anecdotal 

Statistically  significant 
difference  between  the  first 
and  third  maps  was  found. 
Students  liked  the  method 
but  thought  it  was  time 
consuming.  Faculty  thought 
the  maps  demonstrated 
students'  knowledge. 

Contextual 

perspective, 

creativity 

Eerden  (2001) 

Critical  thinking 
vignettes 

To  teach  critical  thinking  skills 
in  a  controlled  laboratory 
environment  that  simulates 
the  complexity  of  clinical 
nursing  situations 

Anecdotal 

Favorable  faculty  response. 

Confidence, 

contextual 

perspective 

Jenkins  & 

Turick-Gibson 

(1999) 

Role  playing 

To  increase  students' 

awareness  of  the  effects  of 

disease  on  the  life  of  some¬ 
one  living  with  the  disease 

Anecdotal 

Students’ journals  showed 
critical  thinking  through 
problem  solving  during 
their  experience. 

Contextual 

perspective, 

creativity 

Jones  & 

Sh  eridan 
(1999) 

Case  study 

To  provide  novice  nurses 
with  a  chance  to  make 
decisions,  apply  knowledge, 
and  examine  beliefs  in  a 
safe  setting 

Learners'  participation 
was  evaluated  during 
the  activity  using  a 
clinical  competence 
checklist  and  a 
qualitative  evaluation 
summary 

Open-mindedness, 

creativity 

Krejci  (1997) 

Imagery 

To  stimulate  students  to  think 
about  the  images  that  appear 
in  response  to  terms  common 
to  nursing 

Anecdotal 

Positive  feedback  from 

students 

Contextual 

perspective, 

open-mindedness 

Mottola  & 
Murphy  (2001) 

“Antidote 
Dilemma,”  an 
interactive  group 
activity  to  solve 
a  nursing  care 
dilemma 

To  teach  students  how  to 
“practice  thinking,"  then  think 
about  and  evaluate  the 
thinking  process 

Anecdotal 

Authors  believed  thinking 
about  factors  students  had 
not  considered  in  their  own 

solution  to  the  dilemma  was 
most  valuable  in  teaching 
critical  thinking. 

Reflection, 

open-mindedness 

Figure  4:  Summary  of  Critical  Thinking  Teaching  Techniques,  Topics  Covered, 
Assessment  Methodologies,  and  Results  in  Nursing  Education 
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TABLE  (Continued) 

Sum  mary  of  Studies  Using  Critical  Thinking  Strategies 


Author 

Method 

Goal 

Evaluation 

Method 

Critical  Thinking 
“Habits  of  the  Mind" 

Results  Addressed 

Perciful  & 

Nester  (1996) 

CAI 

To  determine  the  effect  on 
students'  knowledge  and 
critical  thinking  of  CAI  vs. 
clinical  experience 

Critical  thinking  was 
measured  using  two 
parts  of  the  NLN 
Psychiatric  Nursing 
Examination. 
Knowledge  was 
measured  using  two 
other  parts  of  NLN 
Examination  and  the 
Mosby  AssessTest. 

CAI  students  scored  higher 
on  the  NLN  Examination 
(i.e.,  critical  thinking).  No 
statistically  significant 
difference  in  knowledge 
was  found  between 
the  groups. 

Creativity, 

contextual 

perspective, 

flexibility 

Rossignol 

(1997) 

Discourse 

strategies 

To  compare  effectiveness  of 
five  strategies:  teacher 
high-level  questions, 
teacher  elaboration  of 
student  ideas,  teacher 
probing  questions,  student 
participation,  and  student- 
to-student  participation 

WGCTA  scores 

Inconsistent,  small  positive  — 

relationship  was  found 
between  high-level  teaching 
questions  and  critical  thinking. 

Large  inverse  relationship 
was  found  between  student 
participation  and  critical 
thinking. 

Russaw  (1997) 

Personally 

perceived 

problem 

technique 

To  have  students  identify 
personal  learning  needs, 
work  through  these  needs, 
and  identify  future 
improvements 

Inquisitiveness, 

flexibility, 

creativity 

Saucier  et  at 
(2000) 

CAI  vs.  nursing 
process 

To  compare  the  effect  of  CAI 
and  written  case  studies 
strategies  on  nurse 
practitioner  students’ 
critical  thinking  abilities 

CCTST 

No  significant  increase  in 
critical  thinking  occurred 
using  either  method. 

Creativity, 

flexibility, 

contextual 

perspective 

Schumacher  &  Critical  thinking 
Severson  (1996)  rounds 

To  facilitate  continuing 
development  of  critical 
thinking  by  nurses  and  create 
a  dialogue  among  various 
levels  of  nurses  and  students 

Anecdotal 

Nurses  and  students 
reported  a  “therapeutic” 
effect. 

Open-mindedness, 

creativity 

Sorrell  et  al. 
(1997) 

Interdisciplinary  — 

written  assignments 
and  portfolios 

" 

" 

" 

Todd  (1998) 

E-mailed  multiple- 
choice  questions. 
Students  reported 
their  rationale  for 

selection  of 
correct  and 

incorrect 

responses. 

To  increase  student 
participation  in  learning  and 
increase  student-teacher 

interaction 

Students  completed 
a  questionnaire  at 
the  end  of  the  course 

Most  students  responded 
that  they  felt  the  exercises 
using  e-mail  were  useful 
and  improved  their  critical 
thinking,  and  recommended 
future  use. 

Note:  CAI  =  computer-assisted  instruction;  CCTST  =  California  Critical  Thinking  Skills  Test;  NLN  =  National  League  for  Nursing; 
WGCTA  =  Watson- Glaser  Critical  Thinking  Appraisal. 


Figure  4:  Continued...  Summary  of  Critical  Thinking  Teaching  Techniques,  Topics 
Covered,  Assessment  Methodologies,  and  Results  in  Nursing  Education 

In  a  longitudinal  study  at  a  single  institution,  Catherine  M.  Walsh  and  Lisa  A.  Seldomridge 
analyzed  teaching  strategies  and  results  over  multiple  years.  They  reported: 

Although  demonstrating  modest  gains  in  critical  thinking  dispositions  by  some  cohorts, 
data  collected  from  1997  to  2002  showed  no  consistency  of  pattern.  While  some  cohorts 
saw  gains,  subsequent  cohorts  saw  losses,  and  several  years  had  essentially  no  change. 
Throughout  this  period,  the  usual  academic  indicators  (e.g.,  Scholastic  Aptitude  Test 
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(SAT)/ American  College  Test  (ACT)  data,  cumulative  grade  point  average,  science 
grades)  remained  the  same.  The  data  on  critical  thinking  skills  were  likewise 
inconclusive  and  difficult  to  explain.  This  raised  concerns  about  both  the  definition  of 
critical  thinking  and  the  utility  of  using  standardized  instruments  to  measure  critical 
thinking.178 

Walsh  and  Seldomridge  noted  the  lack  of  correlation  between  performance  on  a  test  and  actual 
job  performance: 

The  use  of  standardized  instruments  to  measure  critical  thinking  skills  is  not  particularly 
useful  because  such  tools  assess  the  skills  of  classical  logic,  as  opposed  to  the  critical 
thinking  skills  of  clinical  practice.  There  is  a  substantial  disconnect  between  critical 
thinking  assessments  and  sought-after  clinical  skills.179 

They  also  highlighted  the  challenge  of  teaching  a  subject  when  it  is  not  well  understood: 

Other  than  fact  acquisition,  pedagogical  interventions  and  strategies  to  boost  critical 
thinking  have  been  largely  unproven,  ineffective,  or  of  unknown  utility  because  we,  as 
nurse  educators,  have  little  understanding  of  the  specific  types  of  reasoning  we  are  trying 
to  cultivate.180 

Summary  of  Lessons  Learned:  The  long  term  analyses  from  the  nursing  profession  and  critical 
thinking  experiments  summarized  as  lessons  learned  in  this  section  clearly  identify  success 
factors  and  obstacles  to  developing  critical  thinking  abilities.  The  factors  that  contribute  to 
effective  teaching  as  discerned  from  the  studies  include: 

•  Clarity  and  agreement  about  what  critical  thinking  is  and  what  students  should  be 
able  to  do 

•  Common  use  of  effective  teaching  strategies  that  focus  on  and  support  the 
definition  of  critical  thinking 

•  Opportunities  for  meaningful  and  extensive  practice  that  reinforce  all  aspects  of 
critical  thinking 

•  Addressing  the  totality  of  the  definition  of  critical  thinking  in  the  program 

•  Including  an  effective  means  of  assessment  to  test  what  had  been  learned 

•  Focus  on  continuous  improvement 
The  factors  that  hinder  effective  teaching  include: 

•  Treating  critical  thinking  as  a  buzzword  -  just  using  the  term  rather  than  explicitly 
teaching  thinking  skills 

•  Not  having  or  not  following  a  clear  definition  and  description  of  critical  thinking; 
not  clarifying  what  a  student  who  thinks  critically  will  be  able  to  do 

•  Using  teaching  strategies  that  are  not  connected  with  critical  thinking,  such  as 
having  students  keep  a  journal,  as  a  way  to  foster  critical  thinking 


178  Walsh,  C.M.  &  Seldomridge,  L.A.  "Critical  Thinking:  Back  to  Square  Two,"  Journal  of  Nursing  Education,  Vol. 
45,  No.  6,  2006,  pp.  212-213. 

179  Walsh  &  Seldomridge,  p.  216. 

180  Walsh  &  Seldomridge,  p.  216. 
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•  Not  using  standard  teaching  strategies  or  letting  instructors  develop  their  own 
strategies  and  approaches 

•  Limited  or  no  opportunity  for  explicit  practice 

•  Inability  to  assess  the  degree  to  which  the  students  met  the  expectations  for  thinking 
critically 

4.3.3.  Critical  Thinking  Training  Courses 

In  this  section  of  the  report,  we  examine  actual  training  courses  designed  for  adult  learners  and 
intended  as  post-college  professional  development  experiences.  I  secured  a  number  of  critical 
thinking  course  descriptions  and  syllabi  in  order  to  evaluate  the  course  content.  These  examples 
include  both  public  courses  with  open  enrollment  and  courses  specifically  developed  for  the 
intelligence  community. 

Public  Courses  with  Open  Enrollment  I  found  multiple  providers  of  public  courses,  each  with 
its  own  model  of  critical  thinking,  differing  topics  of  focus,  and  widely  differing  amounts  of  time 
allotted  to  the  training  experience.  When  comparing  these  courses  against  the  success  factors 
identified  in  the  critical  thinking  training  experiments  section  of  this  report  (Section  4.3.2.),  none 
meet  all  criteria.  A  sample  of  three  public  courses  is  included  in  this  report  to  illustrate  typical 
content  and  teaching  approach. 

Foundation  for  Critical  Thinking  -  The  majority  of  workshops  provided  by  the  Foundation  are 
focused  on  teaching  K-12  teachers  how  to  include  critical  thinking  in  the  classroom.  However, 
they  also  offer  a  series  of  five  two-day  workshops  on  specific  aspects  of  critical  thinking  for 

i  o  1 

business  and  professionals.  The  Foundation  model  for  critical  thinking  is  shown  in  Figure  5. 


181  “The  Critical  Thinking  Community,”  Foundation  for  Critical  Thinking.  Available: 
http://www.criticalthinking.org/starting/index.cfm.  Viewed:  19  April  2010. 
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The  Essential  Dimensions  of  Critical  Thinking 


Our  conception  of  critical  thinking  is  based  on  the  substantive  approach  developed  by  Dr.  Paul  and 
his  colleagues  at  the  Foundation  for  Critical  Thinking  during  the  last  quarter  century.  It  is  relevant  to 
every  subject,  discipline,  profession  and  to  reasoning  through  the  problems  of  every  day  life.  It 
entails  five  essential  dimensions  of  critical  thinking: 

1 .  The  analysis  of  thought 

2.  The  assessment  of  thought 

3.  The  dispositions  of  thought 

4.  The  skills  and  abilities  of  thought 

5.  The  obstacles  or  barriers  to  critical  thought 

Here  (left)  is  an  overview  of  the  first  three  dimensions: 


In  sum,  the  elements  or  structures  of  thought  enable  us  to  "take  our  thinking  apart"  and  analyze  it. 
The  intellectual  standards  are  used  to  assess  and  evaluate  the  elements.  The  intellectual  traits  are 
dispositions  of  mind  which  embody  the  fairminded  critical  thinker.  To  cultivate  the  mind,  we  need 
command  of  these  essential  dimensions,  and  we  need  to  consistently  apply  them  as  we  think 
through  the  many  problems  and  issues  in  our  lives. 


Intellectual  Traits 


Intellectual  Humility  Intellectual  Perseverance 

Intellectual  Autonomy  Confidence  in  Reason 

Intellectual  Integrity  Intellectual  Empathy 

Intellectual  Courage  Fairmindedness 


The  Elements 

Purposes 

Inferences 

Questions 

Concepts 

Points  of  view 

Implications 

Information 

Assumptions 

Critical  Thinkers  Routinely 
Apply  Intellectual  Standards 
To  The  Elements  Of  Reasoning  In 
Order  To  Develop  Intellectual  Traits 


The  Standards 

Clarity 

Precision 

Accuracy 

Significance 

Relevance 

Completeness 

Logicalness 

Fairness 

Breadth 

Depth 

Figure  5:  Foundation  for  Critical  Thinking  Model  of  Critical  Thinking 

The  business  and  professional  course  descriptions  are: 

•  Recognizing  the  Importance  of  Critical  Thinking  in  Business.  This  seminar 
focuses  on  introducing  the  foundations  of  critical  thinking,  including  how  to  take 
your  thinking  apart  and  assess  it  for  quality,  and  how  to  apply  critical  thinking 
concepts  and  principles  to  your  work  on  a  daily  basis. 

•  Using  the  Tools  of  Critical  Thinking  to  Make  Better  Decisions.  This  seminar 
focuses  on  deepening  your  understanding  of  the  concepts  and  principles  of  critical 
thinking  and  applying  them  to  decision-making  at  work. 

•  Understanding  the  Barriers  to  Critical  Thinking.  This  seminar  focuses  on  the 
natural  workings  of  the  human  mind  that  impede  our  ability  (as  humans)  to  reason 
at  the  highest  level  of  quality. 

•  Learning  the  Art  of  Analysis.  This  seminar  expands  on  the  elements  of  reasoning 
that  were  introduced  in  the  first  seminar.  Y ou  will  be  in  engaged  in  many  activities 
focused  on  analyzing  reasoning  in  numerous  everyday  work  situations  -  both  your 
own  reasoning  as  well  as  the  reasoning  of  others. 

•  Learning  the  Art  of  Assessing  Thought.  This  seminar  expands  on  the 
introduction  of  intellectual  standards  for  thought  which  were  introduced  in  the  first 
seminar.  You  will  learn  how  to  apply  these  standards  to  your  thought  and  in  your 
work  at  all  levels.  You  will  also  leam  to  apply  these  standards  to  the  reasoning  of 
others.  182 


182  “The  Critical  Thinking  Community,”  Foundation  for  Critical  Thinking.  Available: 
http://www.criticalthinking.org/professionalDev/business.cfm,  Viewed:  28  April  2011. 
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Of  the  public  courses  featured  in  this  section  of  the  report,  the  courses  presented  by  the 
Foundation  for  Critical  Thinking  appear  to  provide  the  most  depth  and  the  most  opportunities  for 
practicing  critical  thinking  skills. 

HeadS cratchers  LLC  -  HeadScratchers  is  an  organization  specializing  on  providing  critical 
thinking  training  for  business  professionals.  They  offer  one  and  two  day  introductory  critical 

...1 8T 

thinking  workshops.  The  HeadScratchers  critical  thinking  model  is  shown  in  Figure  6. 

We  present  a  simple  framework  for  Critical  Thinking 

ln,om'atlon'  ana  iaeas 


Clarity 

Conclusions 

1 

Decisions 

Figure  6:  HeadScratchers  Model  of  Critical  Thinking 

The  objectives  for  the  foundational  course  are: 

•  Apply  the  three-  step  Critical  Thinking  Process  to  Problem  Solving  and  Decision 
making. 

•  Apply  critical  thinking  to  Get  Clear  on  a  Problem  or  Decision  aided  with  the  Clarity 
Thinksheets 

•  Use  the  Conclusion  Thinksheets  to  guide  and  document  your  thinking  for 
Conclusions 

•  Use  the  Decision  Thinksheets  to  guide  and  document  your  thinking  for  Decisions. 

•  Implement  your  next  step  plan  (created  in  the  workshop)  and  report  how  your 

1 84 

thinking  has  changed. 

Pearson  -  Pearson  now  owns  the  rights  to  the  Watson-Glaser  Critical  Thinking  Assessment 
(WGCTA).  Pearson  offers  two  versions  of  critical  thinking  training.  The  first  is  called  Critical 
Thinking  Boot  Camp,  which  includes  completing  the  WGCTA  prior  to  participating  in  a  two-day 
workshop.  The  second  Pearson  offering  is  called  Critical  Thinking  University.  This  course 
consists  of  a  series  of  18  one -hour  online  sessions.  The  Pearson  model  for  critical  thinking  is 

1 85 

shown  in  Figure  7. 


183  Anonymous.  Critical  Thinking  for  Problem  Solving,  Decision  Making  and  Creativity  [Workshop  Brochure], 
Headscratchers  LLC,  Greenwood  Village,  CO,  [undated]. 

184Anonymous.  Critical  Thinking  for  Problem  Solving,  Decision  Making  and  Creativity’  [Workshop  Brochure], 
Headscratchers  LLC,  Greenwood  Village,  CO,  [undated]. 

185  Watson,  G.  &  Glaser,  E.M.  Watson-Glaser  II  Critical  Thinking  Appraisal  Technical  Manual  and  User's  Guide, 
NCS  Pearson,  Inc.,  San  Antonio,  TX,  2009,  p.  2. 
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Figure  7:  Pearson  Model  of  Critical  Thinking 

The  course  objectives  for  the  two-day  Critical  Thinking  Boot  Camp  are: 

•  Define  critical  thinking 

•  Identify  your  own  critical  thinking  strengths  and  opportunities 

•  Overcome  barriers  to  thinking  critically 

•  Learn  the  critical  questions  to  be  asked  in  any  situation  that  requires  focused 
attention 

•  Utilize  tools  and  techniques  for  accurately  identifying  the  root  cause  of  a  problem, 
strategically  analyzing  complex  issues,  making  effective  decisions,  and  anticipating 
problems  in  implementation 

1 86 

•  Create  a  plan  of  action  to  make  a  tangible,  positive  impact  in  your  organization 

Open  enrollment  classes  are  designed  for  broad  audiences  in  hopes  of  attracting  a  large  number 
of  participants.  They  do  not  include  industry  specific  information  for  any  particular  profession. 
In  the  next  section  of  the  report  I  review  the  classes  developed  for  intelligence  analysis 
professionals. 

Critical  Thinking  Training  in  the  IC:  The  documents  I  reviewed  related  to  critical  thinking 
training  for  intelligence  analysis  included  course  outlines  or  syllabi,  course  manuals,  curriculum 
descriptions,  and  individual  course  descriptions.  The  primary  sources  of  critical  thinking 
training  are  organizations  within  the  intelligence  community.  However,  I  also  reviewed  a 
sample  of  college  courses  if  they  were  part  of  the  curriculum  for  an  intelligence  analysis  degree. 

I  found  the  following  in  the  intelligence  analysis  course  materials: 

•  Little  commonality  between  courses  -  materials,  topics,  and  depth  are  different  in 
different  courses. 

•  A  simplistic  approach  to  critical  thinking;  little  focus  on  cognitive  skills  and 
dispositions;  frequent  mention  of  critical  thinking  without  substance  behind  the 
topic;  or  including  critical  thinking  as  one  of  many  topics. 

•  Writing  about  critical  thinking  and  structured  analytic  techniques  as  though  these 
are  related  or  the  same  thing. 

•  A  strong  focus  on  avoiding  bias,  as  though  the  absence  of  bias  results  in  critical 
thinking. 


1S6Anonymous.  Critical  Thinking  Boot  Camp  [Course  Brochure] ,  Pearson  Education  Inc.  2009,  p.  2. 
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David  Moore  described  a  40-hour  critical  thinking  course  in  Critical  Thinking  and  Intelligence 
Analysis'. 

This  critical  thinking  and  analytic  problem-solving  course  offers  participants  a  chance  to 
learn  a  paradigm  for  critical  thinking  and  critically  explore  different  structured  methods 
of  analysis.  Texts  by  critical  thinking  experts  Richard  Paul  and  Linda  Elder,  and 
structured  analysis  experts  Morgan  Jones  and  Richards  Heuer,  as  well  as  materials 
developed  by  the  instructor,  teach  the  concepts  and  techniques.  Classroom  problems  as 
well  as  operational  examples  (introduced  and  developed  by  the  students  from  their  own 
work)  reinforce  and  help  transfer  what  is  learned  into  the  operational  environment.  A 
final  project  developed  by  student  teams  completes  the  formal  requirements.187 

The  course  objectives  are: 

•  Use  critical  thinking  techniques  to  provide  structure  to  your  analytic  reasoning. 

•  Identify,  describe,  and  employ  14  methods  for  structured  reasoning. 

•  Demonstrate  critical  thinking  proficiency  through  lecture,  classroom  participation, 
and  weekly  homework  assignments. 

•  Complete  a  final  class  assignment  using  a  minimum  of  five  structured  analytic 
methods  presented  in  this  course. 

•  Apply  knowledge  of  critical  thinking  by  using  a  set  of  analytic  tools  designed  to 

1  oo 

hone  your  skills  as  an  analyst. 

Bruce  Bennett  described  the  Defense  Intelligence  Strategic  Analysis  Program  in  Defense 
Intelligence  Agency’s  {DIA)  Analytic  Competency-Based  Training.  The  Strategic  Analysis 
Program  identified  critical  thinking  as  one  of  five  core  competencies.  The  document  described 
four  courses  that  addressed  critical  thinking.  The  two  descriptions  included  here  are 
representative  of  the  course  offerings: 

•  Fundamentals  of  Intelligence  Analysis  (FIA).  FIA  is  a  four-week  in-residence 
course  designed  for  newly-assigned  analysts  in  the  National  Capital  Region. 
Students  enter  a  training  pipeline  that  begins  with  DIA  orientation  (Gateway),  and 
continues  with  Tomorrow's  Intelligence  Professionals,  a  course  designed  to  educate 
new  employees  about  the  basics  of  intelligence,  the  agency,  and  how  it  fits  into  the 
intelligence  and  defense  communities.  Next,  the  Office  of  the  Director  of  National 
Intelligence's  Fundamentals  of  Analysis  provides  a  thorough  orientation  within  the 
IC.  Finally,  FIA  offers  a  hands-on  practical  application  of  critical  thinking, 
collaboration,  structured  analysis,  analytic  writing,  problem  solving,  and  briefing 
skills. 

•  Critical  Thinking  and  Structured  Analysis  (CTSA).  CTSA  is  an  eight-session 
course  designed  to  provide  the  existing  non-Combatant  Command  (COCOM) 
workforce  that  did  not  attend  FIA  with  critical  thinking  skills  and  a  solid 
understanding  of  structured  analytic  methodologies.  This  course  was  designed  to 


187  Moore,  D.T.  Critical  Thinking  and  Intelligence  Analysis,  Joint  Military  Intelligence  College,  Washington,  D.C., 
2006,  p.  98. 

188  Moore,  p.  99. 
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ensure  that  FIA  graduates  and  their  new  skills  would  be  accepted  by  the  existing 
workforce.189 

A  2008  course  titled  Analysis  101  included  a  focus  on  critical  thinking.  The  vision  for  Analysis 
101,  also  called  Introduction  to  Critical  Thinking  and  Structured  Analysis ,190  was  described  as: 

Analysis  101  is  an  unprecedented  program  of  joint  training  in  rigorous  critical  thinking 
and  intelligence  skills  for  all  analysts  during  their  initial  months  in  the  IC. 

This  four-week  course  provides  instruction  on  critical  thinking  for  analytic  success  and 
the  role  of  analysis  in  the  complex,  evolving  world  of  intelligence.  Working  closely  with 
peers  from  other  agencies,  new  analysts  acquire  a  common  set  of  analytic  skills  and  a 
joint  perspective  on  the  intelligence  profession.  The  course  helps  fulfill  one  of  the  Office 
of  the  Director  of  National  Intelligence’s  (ODNI)  central  missions  -  creating  an  integrated 
IC.191 

The  course  goals  are: 

Improve  the  analytic  abilities  of  IC  analysts  by: 

-  Providing  participants  from  across  the  Community  a  common  foundation  of 
critical  thinking  and  analytic  skills 

-  Familiarizing  participants  with  methodologies  and  other  tools  that  facilitate 
critical  thinking  and  problem  solving 

-  Educating  new  analysts  about  how  they  (and  their  respective  organizations)  fit 

1  Q? 

into  the  overall  IC  mission 

The  National  Defense  Intelligence  College  Catalog  2009-2010  stated  that  the  overall  program 
integrates  core  skills  and  competencies  such  as  critical  thinking  throughout  the  curriculum.  The 
description  below  is  representative  of  the  course  offerings. 

MSI  607  Intelligence  Reasoning  and  Analysis  (3:3:0) 

This  course  focuses  on  art  and  science  of  analysis  and  explores  the  methodologies  and 
processes  of  developing  effective  intelligence  analysis.  The  course  orients  itself  on  the 
ODNI  Analytical  Standards  that  the  IC  employ  analysis  of  alternatives,  and  conduct 
effective  professional  collaboration.  Students  will  explore  the  elements  of  logic,  critical 
thinking,  and  argumentation  as  the  fundamental  components  of  assessing  and  estimating 
threats  and  opportunities  in  the  national  security  environment.  Students  will  then 
examine  analytical  methodologies  with  the  goal  of  mitigating  traditional  analytic  pitfalls 
and  enhancing  the  accuracy  of  assessments.  Throughout  the  course,  the  class  will 
explore  the  numerous  organizational  and  ethical  issues  associated  with  improving 
intelligence  analysis  in  today’s  highly  dynamic  and  increasingly  complex  environment.193 


189  Bennett,  B.  "DIA's  Analytic  Competency-Based  Training,"  American  Intelligence  Journal,  Vol.  26,  No.  2, 
2008/2009,  p.  28. 

190  Immerman,  R.H.  "Transforming  Analysis:  The  Intelligence  Community's  Best  Kept  Secret,"  Intelligence  and 
National  Security,  Vol.  26,  No.  2-3,  2011,  p.  168. 

191  Anonymous.  Analysis  101.  Participant  Guide  2008,  p.  3. 

192  Anonymous.  Analysis  101.  Participant  Guide  2008,  p.  4. 

193  Anonymous.  National  Defense  Intelligence  College.  Academic  Year  2009-2010,  National  Defense  Intelligence 
College,  Washington,  D.C.,  [undated],  p.  84. 
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To  provide  additional  examples  of  the  kinds  and  quality  of  courses  available  to  the  intelligence 
community,  I  have  included  a  selection  of  courses  in  support  of  the  National  Air  and  Space 
Intelligence  Center  (NASIC)  at  Wright-Patterson  Air  Force  Base  in  Ohio.  These  courses  include 
professional  development  offered  at  NASIC,  a  course  from  the  Advanced  Technical  Intelligence 
Center  (ATIC),  and  courses  at  Sinclair  Community  College  and  Clark  State  Community  College, 
which  support  the  ATIC  degree  program. 

NASIC  held  a  two-day  course  entitled  Critical  Thinking  in  March  2010.  This  course  had  a 
strong  emphasis  on  the  thinking  skill  of  inference  and  on  argument  evaluation: 

COURSE  OVERVIEW: 

Critical  thinking  is  a  core  skill  of  the  educated  person  and  an  even  more  important  skill 
for  intelligence  analysts.  An  essential  part  of  critical  thinking  is  the  ability  to  evaluate  the 
quality  of  the  support  for  inferences  -  their  own  and  those  of  others.  Much  of  the 
evaluation  of  inferential  support  in  the  intelligence  community  today  comes  down  to  the 
evaluation  of  sources  -  is  the  source  credible,  objective  (unbiased),  etc.?  While  source 
evaluation  is  absolutely  necessary,  even  biased  sources  sometimes  provide  useful 
information  and  apparently  unbiased  ones  sometimes  do  not.  Therefore,  this  course 
narrows  the  scope  of  critical  thinking  to  evaluating  the  support  for  inferences  directly 
given  the  text  provided  regardless  of  the  source.  It  does  not  replace  source  evaluation, 
but  it  is  complimentary  to  it.  194 

COURSE  OBJECTIVES: 

Upon  successful  completion  of  the  course,  students  will  be  able  to: 

a.  Identify  the  elements  of  an  argument  used  to  support  an  inference,  particularly  the 
main  idea  (the  inference  itself)  and  the  evidence  (facts  and  reasoning)  used  to 
support  the  inference. 

b.  Identify  the  assumptions  (warrants)  required  to  use  that  evidence  in  support  of  that 
inference. 

c.  State  alternatives  those  assumptions  and  any  evidence  or  reasons  that  the  alternative 
assumptions  might  be  true. 

d.  Judge  the  overall  quality  of  support  for  the  original  main  idea  based  on  the 
plausibility  of  the  alternative  assumptions 

e.  Modify  the  main  idea,  if  advisable  and  possible,  to  take  any  plausible  alternative 
assumptions  into  account. 

f.  Work  with  others  in  a  skillful  manner  to  arrive  at  a  consensus  for  items  a  through  e. 

g.  Present  the  findings  of  that  team  work  in  a  clear,  concise  and  cogent  manner.195 

ATIC  offers  a  two-day  course  called  Critical  Thinking  Skills  as  one  component  of  the  Analyst 
Boot  Camp.  It  can  also  be  attended  as  a  stand-alone  course. 


194  Anonymous.  Critical  Thinking.  Course  Syllabus,  NASIC,  Wright-Patterson  AFB,  OH,  2010,  p.  1. 
Anonymous.  Critical  Thinking.  Course  Syllabus,  NASIC,  Wright-Patterson  AFB,  OH,  2010,  pp.  2-3. 
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Course  Description 

The  Critical  Thinking  Skills  course  introduces  the  IC  and  other  professionals  to  thought 
and  reasoning  processes.  Students  learn  about  biases,  prejudices,  and  fallacies  which 
hinder  the  thought  process  and  how  to  avoid  them.  This  course  develops  the  necessary 
skills  to  provide  innovative  solutions  to  threats  never  before  encountered. 

What  You  Will  Cover 

•  Conceptualize,  apply,  analyze,  synthesize,  and  evaluate  information 

•  Apply  unbiased  reasoning  and  logic 

•  Analyze  situations  from  both  the  US  and  adversarial  points  of  view 

•  Students  analyze  a  current  event  and  give  a  presentation196 

Sinclair  Community  College  and  Clark  State  Community  College  offer  Certificate  and  Associate 
degree  programs  for  intelligence  analysis.  Neither  school  requires  critical  thinking  coursework 
during  the  certificate  program,  but  both  do  for  the  Associate  degree.  The  critical  thinking 
courses  provided  by  both  institutions  appear  to  be  standard  philosophy-based  logic  courses. 

The  limited  amount  of  practice  usually  provided  in  existing  intelligence  analysis  courses  was 
noted  by  Tecuci  et  al.  in  their  article  describing  a  proposed  tool  for  teaching  critical  thinking 
called  Teaching  Intelligence  Analysts  Critical  Thinking  Skills  (TIACRITIS).  The  authors  did 
not  propose  a  specific  course.  Instead,  they  recommended  incorporating  TIACRITIS  and  its 
textbook  into  existing  training  courses  to  add  effective  practice  opportunities.  In  describing  the 
need  for  this  tool,  the  authors  highlighted  problems  with  existing  training: 

Based  on  our  years  of  experience  in  training  intelligence  analysts,  law  students,  and  high- 
ranking  military  officers,  we  have  arrived  at  the  following  conclusions  related  to  the 
training  of  intelligence  analysts.  The  first  is  that  training  in  the  evidential  reasoning  tasks 
required  in  these  and  other  important  contexts  cannot  be  learned  effectively  just  by 
listening  to  someone  discuss  his/her  own  analyses,  or  just  by  giving  students  lectures  and 
assigned  readings  on  the  topics.  What  is  absolutely  necessary  is  regular  practice 
involving  analyses  of  evidence  using  either  hypothetical  situations  or  examples  drawn 
from  actual  situations.  In  short,  evidential  analysis  is  mastered  best  by  performing 
analyses  contrived  to  illustrate  the  wide  variety  of  subtleties  or  complexities  so  often 
encountered  in  actual  evidential  analyses. 

Our  second  conclusion  is  based  on  our  inspection  of  the  materials  offered  in  several 
courses  for  training  intelligence  analysts.  It  appears  that  analysts  are  so  often  trained  in 
the  production  of  intelligence  analyses  rather  than  upon  the  actual  process  of  analysis 


19(1  Anonymous.  Course  Description  -  Training:  Critical  Thinking  Skills,  Advanced  Technical  Intelligence  Center, 
Beavercreek,  OH,  2010,  p.  1. 

197  “Advanced  Technical  Intelligence  Certificate,”  Clark  State  Community  College.  Available: 
http://www.clarkstate.edu/programdetails.php7ProgranFl20.  Viewed:  22  March  2010  and  “Advanced  Technical 
Intelligence  Certificate  Overview,”  Sinclair  Community  College.  Available: 

http://www. sinclair.edu/components/bulletin/bulletin. cfc?method=displa.  Viewed:  22  March  2010. 

198  “Advanced  Technical  Intelligence,”  Clark  State  Community  College.  Available: 

http://www.clarkstate.edu/programdetails.php7ProgranF4.  Viewed:  22  March  2010  and  “Advanced  Technical 
Intelligence  Degree,”  Sinclair  Community  College.  Available: 

http://www. sinclair.edu/components/bulletin/bulletin. cfc?method=displa.  Viewed:  22  March  2010. 
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itself.  Very  little  training  is  offered  regarding  the  properties,  uses,  discovery,  and 
marshaling  of  the  evidence  upon  which  all  analyses  rest.199 

Some  of  the  course  descriptions  I  reviewed  mentioned  projects  or  case  studies  as  final  activities 
that  would  demonstrate  the  learning  that  took  place  during  the  course.  However,  I  found  no 
information  describing  the  impact  of  the  training  on  the  analysts  as  demonstrated  by  these 
projects  or  on  their  subsequent  analytic  performance. 

The  lack  of  consistent  understanding  of  what  critical  thinking  is  and  how  to  apply  it  is 
demonstrated  in  the  wide  variation  in  course  content,  topics,  depth,  and  specific  skills  taught.  An 
analyst  can  attend  one  of  these  and  believe  that  he  or  she  is  trained  in  critical  thinking,  yet  have  a 
fundamentally  different  understanding  of  critical  thinking  than  attendees  of  other  courses.  If 
critical  thinking  is  as  important  as  it  appears  to  be  from  its  inclusion  in  intelligence  community 
governing  documents,  it  is  important  that  analysts  receive  training  focused  on  common  skills  in 
order  to  develop  common  competencies. 

4.3.4.  Summary:  Teaching  Critical  Thinking  Effectively 

Barry  Beyer  introduced  an  analogy  for  teaching  thinking  skills  in  Practical  Strategies  for  the 
Teaching  of  Thinking.  Since  successfully  teaching  someone  to  play  tennis  (or  any  physical 
activity)  requires  focus  on  the  individual  skills  and  focus  on  how  to  combine  the  skills  to  achieve 
the  outcome  of  winning  the  game,  teaching  critical  thinking  might  take  a  similar  approach. 

Thinking  skills,  on  the  other  hand,  are  very  specific  operations  we  deliberately  perform 
on,  or  with,  data  to  accomplish  our  thinking  goals  -  operations  like  identifying  a  problem, 
finding  unstated  assumptions,  or  assessing  the  strength  of  an  argument.  In  a  way, 
thinking  is  like  playing  tennis  or  any  other  skilled  performance.  Combining  many 
procedures,  it  is  still  greater  in  sum  than  all  these  procedures  combined.  Just  as  tennis 
consists  of  many  specific  skills  -  serving,  making  drop  shots,  lobbing,  and  volleying, 
thinking  too  consists  of  specific  skills,  the  mastery  of  each  of  which  contributes  to 
effectiveness  in  the  entire  process.  Just  as  playing  tennis  requires  the  integration  of 
specific  skills  with  an  overall  strategy  in  a  given  context  for  a  purpose,  so  too  does 
thinking  require  the  integration  of  specific  skills  with  an  overall  strategy  in  a  given 
context  for  a  purpose.  Thinking,  like  tennis  or  any  other  skilled  performance,  is  more 
than  a  bag  of  individual  skills.200 

The  successful  experiments  described  in  Section  4.3.2.  of  this  report  illustrate  what  is  needed  for 
effective  training  in  critical  thinking  (indeed,  for  any  topic): 

1 .  Have  a  clear  desired  outcome  -  what  is  the  increased  critical  thinking  expected  to 
accomplish? 

2.  Have  a  shared,  understandable,  and  comprehensive  definition  of  critical  thinking 

that  will  address  the  desired  outcome  and  includes: 

a.  Thinking  skills 

b.  Thinking  dispositions 

c.  Maturity  of  personal  epistemology 

d.  How  critical  thinking  supports  intelligence  analysis 


199  Tecuci,  G.,  Schum,  D.,  Boicu,  M.,  Marcu,  D.,  Hamilton,  B.  &  Wible,  B.  "Teaching  Intelligence  Analysis  with 
TIACRITIS,"  American  Intelligence  Journal,  Vol.  28,  No.  2,  2010,  pp.  50-51. 

200  R  Beyer,  B.K.  Practical  Strategies  for  the  Teaching  of  Thinking,  Allyn  and  Bacon,  Boston,  MA,  1987,  p.  38. 
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3.  Have  a  well-designed  program  to  develop  the  analyst’s  thinking  skills,  dispositions, 
and  epistemology. 

a.  Topics/contents  (see  Table  1 1  for  the  recommended  skills  and  dispositions) 

b.  Instructional  methodologies 

c.  Opportunities  for  meaningful  practice 

d.  Faculty  well  trained  in  all  of  the  above 

e.  Consistent  implementation  by  all  faculty  involved 

4.  Secure  or  develop  an  assessment  that  will  confirm  students  learned  what  was 
intended  and  achieved  the  desired  outcome. 

5.  Take  corrective  action  to  ensure  program  is  able  to  deliver  what  promised. 

4.4  Assessing  Critical  Thinking 

After  training  someone  to  think  critically,  it  is  important  to  be  able  to  determine  if  the  individual 
learned  enough  to  be  a  critical  thinker.  The  ability  to  assess  someone’s  skills  or  dispositions  has 
long  been  of  interest  to  critical  thinking  scholars.  The  earliest  known  critical  thinking  assessment 
was  available  in  1938.  Over  90  documents  reviewed  for  this  study  addressed  assessments  in 
some  fashion.  The  documents  reviewed  mentioned  the  following  kinds  of  assessments: 

Standardized,  publicly  available  assessments: 

•  Critical  thinking  skills  such  as  the  Cornell  Critical  Thinking  Test  and  the  WGCTA 

•  Critical  thinking  dispositions  such  as  the  CCTDI 

•  General  achievement  or  screening  tests  with  critical  thinking  components  such  as 
Law  School  Admissions  Test  and  the  Collegiate  Learning  Assessment 

Custom-developed  assessments: 

•  Rubrics  -  formal  descriptions  of  expected  outcomes  and  performance  standards 
typically  defined  for  use  in  one  organization  or  institution 

•  Ad  hoc  assessments  such  as  those  identified  in  the  previous  section  for  nursing 
education 

•  Performance-based  assessments  such  as  the  evaluation  implied  in  the  Baker 
University  study 

James  Breckenridge  identified  four  levels  of  assessment  that  can  be  used  to  determine  training 
effectiveness.  The  levels  are: 

•  Level  1 :  Documents  the  participants’  reactions  to  the  training,  usually  with  a 
familiar  questionnaire  asking  “How  satisfied  were  you  with  .  .  .?” 

•  Level  2:  Assesses  students’  knowledge  gained,  typically  through  pre-  and  post¬ 
course  testing. 

•  Level  3:  Evaluates  the  transfer  of  knowledge  to  the  learner’s  work  environment  by 
assessing  behavior  on  the  job  using  reports  from  self,  bosses,  and  peers. 
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•  Level  4:  Assesses  the  training  results  or  outcomes  at  the  organizational  level,  for 

example,  in  quality,  efficiency,  or  customer  satisfaction  statistics;  revenues;  or  some 

20 1 

other  metric  of  strategic  corporate  goal. 

Levels  3  and  4  are  the  most  pertinent  to  our  discussion  regarding  intelligence  analysis.  That  is, 
was  the  student  able  to  apply  what  he  or  she  learned  at  work?  Did  the  new  knowledge  or  skill 
have  a  measurable  impact  on  the  person’s  deliverables?  For  example,  did  students  improve  the 
quality  of  the  intelligence  they  produced? 

Because  the  documents  reporting  assessment  results  were  primarily  focused  on  college  students, 
there  was  little  opportunity  to  assess  level  3  or  4  outcomes.  The  literature  I  found  reported  the 
use  of  a  variety  of  Level  2  evaluation  methods: 

•  Pre-test,  post-test  situations  to  identify  the  amount  of  growth  that  had  been  achieved 
through  some  intervention  to  develop  thinking  skills  or  dispositions. 

•  Post-test  only  to  assess  the  skill  or  disposition  level  of  cohorts  of  individuals. 

•  Post-test  only  for  comparison  among  institutions. 

As  noted  earlier  in  this  report,  the  Baker  University  experiment  implied  a  Level  3  assessment 
because  the  training  was  intended  to  improve  performance  in  the  senior  capstone.  However, 
Baker  did  not  provide  data  on  actual  student  performance  on  the  senior  capstone  project  after  the 
University  provided  the  critical  thinking  training. 

4.4.1,  Standardized  Assessment  Descriptions 

We  were  unable  to  secure  actual  copies  of  the  assessments  listed  below,  but  did  obtain 
descriptions  of  the  assessments  from  their  authors: 

•  Watson  Glaser  Critical  Thinking  Appraisal 

•  CCTST 

•  Halpern  Critical  Thinking  Assessment 

•  CCTDI 

We  secured  copies  of  the  following  assessments: 

•  Cornell  Critical  Thinking  Tests 

•  Ennis-Weir  Critical  Thinking  Essay  Test 

In  addition  to  these  publicly  available  assessments,  a  number  of  scholars  described  assessments 
they  used  in  their  research  such  as  Stanovich’s  Actively  Open-Minded  Thinking  assessment. 
Since  these  assessments  are  not  publicly  available,  they  are  not  addressed  in  this  report. 

Watson-Glaser  II  Critical  Thinking  Appraisal.  Goodwin  Watson  and  Edward  Maynard  Glaser 
first  published  this  assessment  in  1938  and  updated  it  periodically  thereafter.  Pearson,  parent 
company  of  a  variety  of  businesses  including  Pearson  Education  and  Pearson  TalentLens,  now 
owns  the  assessment  and  has  incorporated  it  into  the  Critical  Thinking  Boot  Camp  training 
program  described  previously  in  this  report.  From  the  introduction  of  the  appraisal: 


201  Breckenridge,  J.G.  "Designing  Effective  Teaching  and  Learning  Environments  for  a  New  Generation  of 
Analysts,"  International  Journal  of  Intelligence  and  Counterintelligence,  Vol.  23,  No.  2,  2010,  p.  308. 
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Watson  and  Glaser  (Glaser,  1937;  Watson  &  Glaser,  1994)  believed  that  critical  thinking 
includes: 

•  attitudes  of  inquiry  that  involve  an  ability  to  recognize  the  existence  of  problems 
and  an  acceptance  of  the  general  need  for  evidence  in  support  of  what  is  asserted  to 
be  true, 

•  knowledge  of  the  nature  of  valid  inferences,  abstractions,  and  generalizations  in 
which  the  weight  or  accuracy  of  different  kinds  of  evidence  are  logically 
determined,  and 

202 

•  skills  in  employing  and  applying  the  above  attitudes  and  knowledge.  ' 

Consistent  with  this  conceptualization,  the  Watson-Glaser  II  has  maintained  the  same 
approach  to  measuring  critical  thinking.  Each  Watson-Glaser  II  subtest  is  composed  of 
reading  passages  or  scenarios  that  include  problems,  statements,  arguments,  and 
interpretations  of  data  similar  to  those  encountered  on  a  daily  basis  at  work,  in  the 
classroom,  and  in  newspaper  or  magazine  articles.  Each  scenario  is  accompanied  by  a 

90'} 

number  of  items  to  which  the  participant  responds. 

CCTST:  Peter  Facione  developed  this  assessment  based  on  the  APA  consensus  definition  of 
critical  thinking.  The  web  page  from  the  publisher  of  this  assessment  contains  the  following 
description: 

The  CCTST  Family  is  a  set  of  critical  thinking  skills  tests  based  on  the  Delphi  Expert 
Consensus  Definition  of  Critical  Thinking.  Research  has  shown  that  these  instruments 
predict  strength  in  critical  thinking  in  authentic  problem  situations  and  success  on 
professional  licensure  examinations.  As  measures  of  core  critical  thinking  skills,  these 
tools  address  the  application  of  one’s  reasoning  skills  for  the  purpose  of  forming  a 
reflective  judgment  about  what  to  believe  or  what  to  do  in  a  given  context  or  problematic 
situation. 

All  the  tests  listed  below  in  the  CCTST  family  provide  objective  measures  of  critical 
thinking  skills.  Different  questions  progressively  invite  test-takers  to  analyze  or  to 
interpret  information  presented  in  text,  charts,  or  images;  to  draw  accurate  and  warranted 
inferences;  to  evaluate  inferences  and  explain  why  they  represent  strong  reasoning  or 
weak  reasoning;  or  to  explain  why  a  given  evaluation  of  an  inference  is  strong  or  weak.204 

Halpern  Critical  Thinking  Assessment:  Diane  Halpem  defined  the  Halpern  Critical  Thinking 
Assessment  (HCTA)  in  order  to  achieve  some  balance  between  the  ease  of  scoring  a  multiple 
choice  assessment  with  the  need  for  asking  for  open-ended  responses.  Halpern’s  web  page 
contains  a  description  of  the  assessment  and  links  to  sample  questions. 

Unlike  other  tests  of  critical  thinking,  the  HCTA  uses  both  open  ended  and  forced  choice 
questions.  Both  response  formats  have  advantages  and  limitations.  Forced  choice 
questions  assess  the  ability  to  recognize  a  correct  response,  but  there  are  few  instances  in 
real  life  where  people  are  presented  with  an  array  of  answers  to  select  from.  Recognition 
is  a  lower-level  cognitive  skill,  which  is  expected  to  yield  higher  estimates  of  critical 


202  Watson,  G.  &  Glaser,  E.M.  Watson-Glaser  II  Critical  Thinking  Appraisal  Technical  Manual  and  User's  Guide, 
NCS  Pearson,  Inc.,  San  Antonio,  TX,  2009,  p.  2. 

203  Watson  &  Glaser,  p.  2. 

204  “California  Critical  Thinking  Skills  Tests,”  Insight  Assessment.  Available:  http://www.insightassessment.com/. 
Viewed:  29  April  2010,  p.  2. 
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thinking  skill  than  constructed  response  questions,  which  requires  higher-level  cognitive 
processing.  The  disadvantage  of  constructed  response  questions  is  that  they  benefit 
people  with  good  writing  skills,  and  thus  may  underestimate  the  critical  thinking  skills  of 
mediocre  writers.  There  is  evidence  that  multiple-choice  and  open-ended  responses  are 
measuring  separate  cognitive  abilities  (Bridgeman  &  Moran,  1997).  The  constructed 
response  portion  of  the  HCTA  attempts  to  reveal  more  of  the  dispositional  component  of 
thinking,  as  it  allows  test-takers  to  demonstrate  whether  they  are  inclined  to  apply  the 
appropriate  skills  (Ku,  2009). 205 

CCTDI:  Peter  and  Noreen  Facione  developed  this  assessment  based  on  the  APA  consensus  list 
of  dispositions.  This  is  the  only  assessment  that  deals  specifically  with  dispositions  rather  than 
skills.  The  web  page  from  the  publisher  of  this  assessment  contains  the  following  description: 

There  are  seven  scales  on  the  CCTDI.  Each  describes  an  aspect  of  the  overall  disposition 
toward  using  one's  critical  thinking  to  form  judgments  about  what  to  believe  or  what  to 
do.  People  may  be  positively,  ambivalently,  or  negatively  disposed  on  each  of  seven 
aspects  of  the  overall  disposition  toward  critical  thinking.  The  CCTDI  also  provides  a 
Total  score  which  gives  equal  weight  to  each  of  the  seven. 

•  Truthseeking 

•  Open-mindedness 

•  Analyticity 

•  Systematicity 

•  Critical  Thinking  Self-Confidence 

•  Inquisitiveness 

•  Maturity  of  Judgment  206 

Cornell  Critical  Thinking  Tests:  Robert  Ennis  and  Jason  Millman  developed  this  assessment. 
The  introduction  to  the  manual  associated  with  the  assessment  explains  the  conceptual 
foundation  for  the  tool: 

The  working  definition  of  "critical  thinking"  under  which  we  are  operating  is  this: 
"Critical  thinking  is  reasonable  and  reflective  thinking  focused  on  deciding  what  to 
believe  or  do."  This  definition  is  an  attempt  to  capture  the  main  thrust  of  the  term  'critical 
thinking'  as  it  is  used  these  days. 

There  are  many  ways  to  dissect  and  subcategorize  critical  thinking  ability.  Our  approach 
sees  three  types  of  inferences  to  beliefs  (induction,  deduction,  and  value  judging);  and 
four  types  of  bases  for  such  inferences,  which  are:  1)  the  results  of  other  inferences,  2) 
observations,  3)  statements  made  by  others,  and  4)  assumptions.  Furthermore,  close 
attention  to  meaning  must  permeate  one's  dealing  with  the  three  types  of  inferences  and 
four  types  of  bases.207 


205  Halpern,  D.F.  “Halpern  Critical  Thinking  Assessment  (HCTA),”  Schufried  Publishers.  Available: 
http://www.cmc.edu/pages/faculty/dhabem/index  files/Pagel 667.html,  Viewed:  February  7,  2012. 

206“California  Critical  Thinking  Disposition  Inventory:  Scale  Descriptions,”  Insight  Assessment.  Available: 
http://www.insightassessment.com/Scales%20CCTDI.html,  Viewed:  October  9,  2009. 

207  Ennis,  R.H.,  Millman,  J.  &  Tomko,  T.N.  Cornell  Critical  Thinking  Tests  Level  X &  Level  Z  Manual,  5th  ed..  The 
Critical  Thinking  Co.,  Seaside,  CA,  2005,  pp.  1-2. 
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There  are  two  versions  of  the  assessment:  Level  X  for  students  in  grades  4-14  and  Level  Z  for 
advanced  and  gifted  high  school  students,  college  and  graduate  students,  and  other  adults. 

Ennis-Weir  Critical  Thinking  Essay  Test:  Robert  Ennis  and  Eric  Weir  developed  this  essay- 
based  assessment  as  an  alternative  to  multiple  choice  assessments.  The  introduction  explains  the 
purpose  of  the  tool: 

The  Ennis-Weir  is  a  general  test  of  critical  thinking  ability  in  the  context  of 
argumentation.  This  type  of  context  is  one  in  which  someone  is  trying  to  defend  a  point, 
and  in  which  the  defense  is  usually  preceded  and  succeeded  by  other  argumentation  on 
the  point  or  aspects  of  it.  In  this  test,  a  complex  argument  is  presented  to  the  test  taker, 
who  is  asked  to  formulate  another  complex  argument  in  response  to  the  first.  The  test  is 
intended  to  help  evaluate  a  person's  ability  to  appraise  an  argument  and  to  formulate  in 
writing  an  argument  in  response,  thus  recognizing  a  creative  dimension  in  critical 
thinking  ability.208 

Table  12  contains  a  summary  of  the  primary  focus  of  each  assessment.  Although  I  found  a 
number  of  writers  who  provided  opinions  on  the  quality  of  one  or  more  of  these  assessments,  I 
found  no  reports  comparing  the  outcomes  or  effectiveness  of  these  assessments.  I  found  no 
information  proving  the  validity  of  one  assessment  over  another. 


208  Ennis,  R.H.  &  Weir,  E.  The  Ennis-Weir  Critical  Thinking  Essay  Test,  Midwest  Publications,  Pacific  Grove  CA, 
1985,  p.  1. 
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Table  12:  Comparison  of  Standardized  Critical  Thinking  Assessments 


Watson-Glaser 

Cornell  Critical 

Critical  Thinking 

Thinking  Test: 

Appraisal: 

Level  X: 

• 

Inference 

• 

Inductive 

• 

Recognition  of 

Inference 

Assumptions 

• 

Credibility  of 

• 

Deduction 

Sources  and 

• 

Interpretation 

Observation 

• 

Evaluation  of 

• 

Deduction 

Arguments 

• 

Assumption 

Identification 

Level  Z: 

• 

Deduction 

• 

Meaning 

• 

Credibility 

• 

Inductive 

Inference 

• 

Definition  and 
Unstated  Reasons 

• 

Assumption 

Identification 

The  Ennis-Weir  Critical 
Thinking  Essay  Test: 

Getting  the  point 
Seeing  the  reasons  and 
assumptions 
Stating  one's  point 
Offering  good  reasons 
Seeing  other  possibilities 
Responding  appropriately 
to  and/or  avoiding: 

+  Equivocation 
+  Irrelevance 
+  Circularity 
+  Reversal  of  an  "if- 
then"  (or  other 
+  conditional) 
relationship 
+  The  straw  person 
fallacy 

+  Overgeneralization 
+  Excessive  skepticism 
+  Credibility  problems 
+  The  use  of  emotive 
language  to  persuade 


Halpern  Critical 

Thinking  Assessment 

*  verbal  reasoning 
(e.g.,  recognizing 
the  use  of  pervasive 
or  misleading 
language), 

*  argument  analysis 
(e.g.,  recognizing 
reasons  and 
conclusions  in 
arguments), 

*  hypothesis  testing 
(e.g.,  understanding 
sample  size, 
generalizations), 

*  using  likelihood 
and  uncertainty 
(e.g.,  applying 
relevant  principles 
of  probability,  base 
rates), 

*  decision  making 
and  problem 
solving  (e.g., 
identifying  the 
problem  goal, 
generating  and 
selecting  solutions 
among 

alternatives). _ 


CCTST: 

Analysis 

Interpretation 

Inference 

Evaluation 

Explanation 


California  Critical 
Thinking 
Dispositions  Test 

•  Truthseeking 

•  Open-mindedness 

•  Analyticity 

•  Systematicity 

•  Critical  Thinking 
Self-Confidence 

•  Inquisitiveness 

•  Maturity  of 
Judgment 
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A  number  of  organizations  used  customized  assessments  rather  than  the  standardized 
assessments.  These  are  introduced  in  the  next  section. 

4.4.2.  Customized  Assessments 

An  organization  develops  a  customized  assessment  when  it  has  specific  attributes  or  outcomes 
that  it  wants  to  evaluate.  Customized  assessments  can  be  tailored  for  the  specific  needs  of  the 
organization  and  its  unique  competencies  and  outcomes.  This  report  takes  a  closer  look  at  two 
types  of  customized  assessments: 

•  Rubrics 

•  Performance  assessments 

Rubrics'.  A  rubric  is  an  assessment  tool  that  defines  expected  outcomes  and  performance 
standards.  Once  the  rubric  is  defined,  it  can  be  used  to  guide  instructional  content,  inform 
students  of  expected  outcomes,  and  assess  accomplishments. 

William  Pierce  described  the  value  of  rubrics  and  provided  several  samples: 

Professors  who  teach  thinking  skills  such  as  arguing,  analyzing,  synthesizing,  drawing 
conclusions,  solving  problems,  making  decisions,  and  evaluating  need  to  know  how  well 
their  students  can  use  these  skills.  Using  rubrics  that  describe  several  different  levels  of 
student  performance 

•  helps  professors  evaluate  consistently  and  efficiently 

•  lets  students  know  what  their  professor  is  looking  for  and  how  to  meet  the 
expectations 

•  provides  feedback  to  students.209 

Figure  8  contains  one  of  the  samples  provided  by  Pierce.210 


209  Peirce,  W.  “Designing  Rubrics  for  Assessing  Higher  Order  Thinking,”  Available: 

http://academic.pgcc.edu/~wpeirceAfCCCTR/Designingrubricsassessingthinking.html,  Viewed:  7  February  2012. 

210  Peirce,  W.  “Designing  Rubrics  for  Assessing  Higher  Order  Thinking,”  Available: 

http://academic.pgcc.edu/~wpeirce/MCCCTR/Designingmbricsassessingthinking.html,  Viewed:  7  February  2012. 
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Rubric 

Component 

4 

3 

2 

1 

Identifies  and 
summarizes  the 
problem/question 
at  issue. 

Accurately  identities  the 
problem/question  and 
provides  a 
well-developed 
summary. 

Accurately  identities 
the 

problem/question  and 
provides  a  brief 
summary. 

Identifies  the  problem/ 
question  and  provides  a 
poor  summary  or 
identifies  an 
inappropriate 
probl  em/ quest!  on. 

Docs  not  identify  or 
summarize  the 
problem/question 
accurately  if  at  all. 

Identifies  and 

assesses  the 
quality  of 
supporting 
data/evidence 

Provides  a 
well-developed 
examination  of  the 
evidence  and  questions  its 
acciracy,  relevance,  and 
completeness.  Clearly 
distinguishes  between  fact 
and  opinion 

Examines  evidence 
and 

questions  the  quality. 
Disliuguislies  between 
fact  and  opinion. 

Merely  repeats 
information 
provided.  Does  not 
justify  position  or 
distinguish  between  tact 
and  opinion 

Does  not  identify  or 
assess  die  quality  of 
supporting  evidence. 

Figure  8:  Mid-South  Community  College  Critical  Thinking  Rubric 

Diane  Kelly-Riley  et  al.  wrote  about  a  critical  thinking  rubric  used  at  Washington  State 
University  (WSU): 

We  use  the  rubric  as  an  instructional  guide  and  as  an  evaluative  tool  using  a  6-point  scale 
for  evaluation  combining  holistic  scoring  methodology  with  expert-rater  methodology 
(Haswell.  &  Wyche,  1996;  Haswell,  1998).  Early  studies  conducted  by  CTLT  and  the 
Writing  Programs  indicated  an  atmosphere  ready  for  implementation  of  a  critical  thinking 
rubric  within  the  WSU  curriculum. 

The  instrument  itself  identifies  seven  key  areas  of  critical  thinking.  The  dimensions 
include 

•  problem  identification 

•  the  establishment  of  a  clear  perspective  on  the  issue 

•  recognition  of  alternative  perspectives 

•  context  identification 

•  evidence  identification  and  evaluation 

•  recognition  of  fundamental  assumptions  implicit  or  stated  by  the  representation  of 

an  issue,  and 

211 

•  assessment  of  implications  and  potential  conclusions. 

212 

Figure  9  shows  a  sample  portion  of  the  Washington  State  rubric. " 


211  Kelly-Riley,  D.,  Brown,  G.,  Condon,  B.  &  Law,  R.  Washington  State  University  Critical  Thinking  Project,  The 
Center  for  Teaching,  Learning,  Technology,  General  Education  &  The  Writing  Programs,  Pullman,  WA,  2001,  pp. 
2-3. 

212  Kelly-Riley  et  al.,  p.  9. 
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1)  Identifies  and  summarizes  the  problem/question  at  issue  (and/or  the  source’s  position). 


Scant 


Substantially  Developed 


Does  not  identify  and  summarize  the 
problem,  is  confused  or  identifies  a 
different  and  inappropriate  problem. 


Does  not  identify  or  is  confused  by  the 
issue,  or  represents  the  issue  inaccurately. 


Identifies  the  main  problem  and  subsidiary, 
embedded,  or  implicit  aspects  of  the 
problem,  and  identifies  them  clearly, 
addressing  their  relationships  to  each  other. 

Identifies  not  only  the  basics  of  the  issue, 
but  recognizes  nuances  of  the  issue. _ 


2)  Identifies  and  presents  the  STUDENT'S  OWN  perspective  and  position  as  it  is  important  to  the 
analysis  of  the  issue. 


Scant  Substantially  Developed 


Addresses  a  single  source  or  view  of  the 
argument  and  fails  to  clarify  the  established 
or  presented  position  relative  to  one’s  own. 
Fails  to  establish  other  critical  distinctions. 

Identifies,  appropriately,  one's  own 
position  on  the  issue,  drawing  support  from 
experience,  and  information  not  available 
from  assigned  sources. 

3)  Identifies  and  considers  OTHER  salient  perspectives  and  positions  that  are  important  to  the 
analysis  of  the  issue. 

Scant  Substantially  Developed 


Deals  only  with  a  single  perspective  and 
fails  to  discuss  other  possible  perspectives, 
especially  those  salient  to  the  issue. _ 


Addresses  perspectives  noted  previously, 
and  additional  diverse  perspectives  drawn 
from  outside  information. 


Figure  9:  WSU  Critical  Thinking  Rubric 


Performance  Assessments:  Performance  assessments  provide  evaluation  at  the  Breckenridge’s 
Level  3  or  Level  4.  This  level  of  evaluation  is  summarized  by  two  questions:  can  the  individual 
transfer  the  knowledge  to  the  work  environment  and  does  this  transfer  of  knowledge  actually 
improve  performance? 


Rubrics  can  be  used  in  assessing  performance.  However,  I  found  nothing  in  the  literature  that 
reported  on  perfonnance  assessments  using  rubrics. 


I  found  only  one  study  comparing  critical  thinking  ability  with  actual  performance  in  a  project  or 
in  work  as  part  of  the  assessment  literature.  The  nursing  literature  reported  on  a  study  comparing 
critical  thinking  competence  with  clinical  perfonnance  and  found  no  conelation.  The  authors 
did  report  a  conelation  between  years  of  nursing  and  competence. 


Question  3:  Is  there  a  relationship  between  critical  thinking  ability  and  level  of 
competence  as  defined  by  Benner's  stages  of  skill  acquisition  and  measures  on  the  6-D 
Scale?  Survey  data  included  self-rating  responses  for  stage  of  skill  acquisition  and 
measures  on  the  6-D  Scale.  Critical  thinking  measurement  as  a  professional  nurse  was 
collected  (WGCTA-3)  and  compared  to  perceived  level  of  competence  as  measured  by 
Benner's  stages  and  the  6-D  Scale.  Multiple  regression,  analysis  of  variance  and 
correlational  techniques  were  used.  Analysis  was  performed  on  the  subsample  of  30. 
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Findings  revealed  no  significant  effect,  i.e.,  critical  thinking  ability  had  no  influence  on 

^  1 1 

measures  of  nursing  competence.  The  correlation  matrix  (N=24  complete  cases). 

Baker  University  implied  a  perfonnance  objective  when  they  specified  that  the  outcome  of  their 
program  was  to  improve  competence  on  the  senior  capstone  project.  They  did  not,  however, 
explicitly  report  on  the  results  of  their  efforts  to  improve  student  performance  on  this  project. 

4.4.3.  Assessments  in  the  Intelligence  Analysis  Literature 

Only  two  documents  in  the  intelligence  literature  mentioned  critical  thinking  assessments. 

Fischl  and  Gilbert  reported  on  one  additional  application  of  an  assessment  in  1983:  its  use  in 
hiring.  They  tested  whether  the  WGCTA  would  be  useful  in  hiring  intelligence  analysts;  they 
concluded  that  it  would  be  beneficial.  Although  they  theorized  that  the  WGCTA  might  be  useful 
in  hiring,  I  found  no  follow  up  studies  that  evaluated  whether  individuals  hired  using  the 
proposed  assessments  actually  perfonned  better  as  analysts. 

The  three-test  battery  consisting  of  the  Watson-Glaser  Critical  Thinking  Appraisal,  and 
the  Memory  and  Expression  subtests  from  the  series  of  Flanagan  Industrial  Tests, 
requires  about  one  hour  for  administration  and  is  scorable  by  a  clerk  using  stencil 
overlays. 

On  the  basis  of  the  investigation  performed,  it  appears  that  a  relatively  short  battery  of 
easily  administered  and  scored  tests  can  appreciably  improve  the  procedure  for  selection 

214 

of  intelligence  analysts. 

David  Moore’s  book  Critical  Thinking  and  Intelligence  Analysis  mentioned  the  possibility  of 
providing  compensation  for  mastery  of  critical  thinking  skills,  and  noted  that  mastery  would 
need  to  be  assessed.  The  assessment  he  references  is  an  assessment  developed  as  part  of  the 
admissions  process  to  Cambridge  University. 

If  an  analyst  adopts  a  congeries  of  skills  that  contributes  to  the  mastery  of  critical 
thinking,  and  is  compensated  monetarily,  that  mastery  needs  to  be  certified.  If  a 
curriculum  that  drives  the  acquisition  of  those  skills  is  in  place,  then  an  assessment  of 
those  skills  can  be  administered  in-house.  Specific  tests  exist  for  the  assessment  of 

215 

critical  thinking,  such  as  the  "Thinking  Skills  Assessment  Test."' 

Moore  also  reported  that  the  DIA  tested  some  of  its  analysts  using  the  WGCTA,  but  provided  no 

716 

information  on  the  findings  or  results  of  the  assessment. 

The  lack  of  attention  in  the  literature  to  critical  thinking  assessments  may  represent  an  omission 
within  the  intelligence  community’s  body  of  knowledge. 


213  Maynard,  C.A.  "Relationship  of  Critical  Thinking  Ability  to  Professional  Nursing  Competence,"  Journal  of 
Nursing  Education,  Vol.  35,  No.  1,  1996,  p.  16. 

214  Fischl,  M.A.  &  Gilbert,  A.C.F.  Selection  of  Intelligence  Analysts,  ADP000869,  Army  Research  institute  for  the 
Behavioral  and  Social  Sciences,  Alexandria,  VA,  1983,  p.  389. 

215  Moore,  D.T.  Critical  Thinking  and  Intelligence  Analysis,  Joint  Military  Intelligence  College,  Washington,  D.C., 
2006,  p.  70. 

216  Moore,  p.  65. 
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4.4.4.  Summary:  Using  Assessments  Effectively 

The  variety  of  assessments  and  assessment  approaches  that  exist  provide  multiple  methods  to 
determine  the  extent  that  an  individual  learned  something  from  training  and  the  extent  that  he  or 
she  is  able  to  apply  that  learning  to  the  job.  However,  before  selecting  or  developing  an 
assessment,  the  intelligence  community  must  first  detennine  what  definition  or  description  of 
critical  thinking  supports  intelligence  analysis,  how  critical  thinking  fits  within  intelligence 
analysis,  what  specific  skills  and  dispositions  are  needed,  and  how  these  might  be  demonstrated. 
Once  these  elements  are  determined,  an  evaluation  of  standardized  assessments  can  take  place  to 
determine  their  relevance  and  value.  If  the  standardized  assessments  prove  to  be  inadequate,  the 
intelligence  community  can  commission  or  develop  customized  assessments. 

A  definitive  answer  regarding  the  usefulness  of  an  assessment  in  determining  whether  an 
individual  has  the  right  thinking  abilities  or  dispositions  in  order  to  predict  performance  as  an 
intelligence  analyst  does  not  yet  exist. 

Because  performance  assessments  address  actual  on  the  job  perfonnance,  they  are  far  more 
important  than  the  Level  1  or  Level  2  assessments  that  are  the  primary  topics  in  the  literature  and 
the  focus  of  standardized  assessments.  If  they  were  available,  performance  assessments  of 
critical  thinking  in  the  intelligence  community  would  provide  insight  into  whether  or  not  critical 
thinking  can  improve  analysis.  They  might  also  help  determine  which  teaching  approaches  are 
most  effective. 

The  use  of  assessments  that  measure  skill  attainment  at  Levels  3  or  4  is  a  topic  that  would  benefit 
from  additional  research  in  the  intelligence  community.  Without  some  indication  that  critical 
thinking  improves  analyst  performance  and  analytic  results,  little  reason  exists  for  investing  in 
the  development  of  critical  thinking  competency  in  intelligence  analysts. 
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4.5  Critical  Thinking  in  the  Intelligence  Analysis  Literature 

So  far  in  this  report  we  have  explored  what  critical  thinking  is,  the  enablers  of  critical  thinking, 
how  to  train  someone  to  think  critically,  and  how  to  determine  whether  someone  is  able  or 
willing  to  think  critically.  Although  I  have  included  intelligence  analysis  examples,  the  primary 
focus  has  been  on  these  topics  across  all  industries.  This  section  of  the  report  focuses 
exclusively  on  critical  thinking  in  the  intelligence  analysis  literature.  Topics  include: 

•  History  of  critical  thinking  in  the  intelligence  literature 

•  Definitions  of  critical  thinking  for  intelligence  analysis 

•  How  to  apply  critical  thinking  in  intelligence  analysis 

•  Potential  next  steps  to  improve  understanding  of  how  critical  thinking  might 
enhance  the  quality  of  intelligence  analysis  in  the  future 

The  reports  on  critical  thinking  have  given  little  attention  to  additional  ingredients  for  effective 
intelligence  analysis  such  as  having  appropriate  knowledge  or  having  an  appropriate 
methodology  for  conducting  analysis  as  was  defined  in  Framework  for  Analytic  Cognition .217 
Critical  thinking  might  be  a  necessary  and  important  part  of  effective  analysis,  but  it  is  not 
sufficient  by  itself  to  guarantee  high  quality  analysis. 

4.5.1.  A  Brief  History 

Although  the  concept  of  critical  thinking  is  currently  recognized  as  an  important  part  of 
successful  intelligence  analysis,  the  term  was  not  as  widely  used  in  the  past  as  it  is  today. 
Despite  the  public  interest  in  educating  people  to  think  critically  that  began  in  the  1980s  and 
accelerated  when  critical  thinking  became  a  national  educational  goal  in  1990,  the  IC  displayed 
limited  interest  in  the  subject  until  the  21st  Century.  I  found  a  total  of  128  documents  that  had  at 
least  one  mention  of  critical  thinking  with  respect  to  intelligence  analysis.  Table  13  contains  a 
summary  of  these  documents  by  year. 


217  tn  press 


86 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-2014-1598,  Cleared  11  April  2014 


Table  13:  Critical  Thinking  in  Intelligence  Analysis  Literature  by  Year 


Year 

#  of  Documents 

1983 

1 

1994 

1 

1995 

1 

1996 

1 

1997 

1 

1998 

2 

1999 

1 

2000 

2 

2001 

2 

2002 

5 

2003 

8 

2004 

3 

2005 

12 

2006 

9 

2007 

8 

2008 

23 

2009 

10 

2010 

22 

2011 

5 

Undated 

11 

Total 

128 

Some  documents  described  activities  or  characteristics  that  would  be  perceived  today  as 
pertaining  to  critical  thinking  without  mentioning  the  term  itself.  For  example,  David  Schum 
clearly  described  critical  thinking  activities  in  his  1987  book,  Evidence  and  Inference  for  the 
Intelligence  Analyst  although  the  term  critical  thinking  is  not  used  anywhere  in  the  document. 
Rather,  Schum  uses  terms  such  as  inference,  reasoning,  and  logic  to  describe  what  is  now  called 
critical  thinking.  David  Moore’s  2002  thesis  Creating  Intelligence:  Evidence  and  Inference  in 
the  Analysis  Process  follows  Schum’ s  model  of  describing  critical  thinking  without  actually 
using  the  term.  The  book  Psychology  of  Intelligence  Analysis  by  Richards  Heuer  also  included 
activities  that  could  be  considered  critical  thinking,  yet  Heuer’ s  only  mention  of  critical  thinking 
is  to  state  that  it  does  not  mix  well  with  idea  generation  and  imagination.220 

The  earliest  mention  I  found  of  the  term  critical  thinking  in  intelligence  analysis  appeared  in  a 

99  1 

1983  document  describing  the  use  of  a  critical  thinking  assessment  in  personnel  selection." 


218  Schum,  D.A.  Evidence  and  Inference  for  the  Intelligence  Analyst.  Volume  1.  University  Press  of  America, 
Lanham,  MD,  1987. 

219  Moore,  D.T.  Creating  Intelligence:  Evidence  and  Inference  in  the  Analysis  Process,  MSSI  Thesis,  Joint  Military 
Intelligence  College,  2002. 

~2I)  Heuer,  R.J.J.  Psychology  of  Intelligence  Analysis,  Center  for  the  Study  of  Intelligence,  Washington,  D.C.,  1999, 
p.  69. 

221  Fischl,  M.A.  &  Gilbert,  A.C.F.  Selection  of  Intelligence  Analysts,  ADP000869,  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences,  Alexandria,  VA,  1983. 
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Even  when  critical  thinking  was  mentioned,  authors  rarely  included  any  explanation  of  what  was 
meant  by  the  term.  For  example,  the  1995  CIA  documents  Compendium  of  Analytic  Tradecraft 
Notes  includes  references  to  critical  thinking,  primarily  focused  on  how  critical  thinking  is 
simply  a  part  of  the  process: 

When  analysts  address  uncertainty  -  matters  that  require  interpretations  and  estimates 
that  go  well  beyond  the  hard  evidence  -  their  argumentation  must  clarify  the  premises, 
suppositions,  and  other  elements  of  critical  thinking  that  underlie  the  judgments.222 

A  1996  Joint  Military  Intelligence  College  Occasional  Paper  noted  the  lack  of  critical  thinking  in 
analyst  training: 

Further  exacerbating  the  problems  that  come  with  segmentation  is  the  perception  that 
doctrine  can  become  an  undesirable  mental  straight-jacket.  The  potential  value  of 
doctrine  in  Joint  Pub  2-0,  Joint  Doctrine  for  Intelligence  Support  to  Operations,  is 
weakened  because  it  actually  promotes  intuition  over  reason,  as  no  emphasis  is  placed  on 
analytical  skills.  Technical  training  is  based  on  doctrine,  and  the  technical  training  for 
intelligence  personnel  therefore  includes  little  in  managerial,  reasoning,  and  critical 
thinking  skills.223 

David  Moore  and  Fisa  Krizan  explicitly  identified  critical  thinking  as  an  important  contributor  to 
intelligence  analysis  in  2001  in  their  description  of  potential  core  competencies  (characteristics, 
knowledge,  skills,  and  abilities)  for  intelligence  analysts.  ~  They  identified  thinking  as  one  of 
the  needed  abilities,  and  critical  thinking  as  a  needed  skill.  In  2003,  Moore  and  Krizan  expanded 
their  viewpoint  in  “Core  Competencies  for  Intelligence  Analysis  at  the  National  Security 
Agency,”225  which  included  the  same  emphasis  on  thinking  and  critical  thinking  and  the  same 
definition  provided  above.  They  emphasized  that  their  list  of  competencies,  reproduced  in 
Figure  10,  was  a  proposal  for  consideration  rather  than  an  approved  list. 226 


222  Central  Intelligence  Agency.  A  Compendium  of  Analytic  Tradecraft  Notes  1995,  p.  7. 

223  Brei,  W.S.  Getting  Intelligence  Right:  The  Power  of  Logical  Procedure,  Joint  Military  Intelligence  College, 
Washington,  D.C.,  1996,  p.  1. 

224  Moore,  D.T.  &  Krizan,  L.  "Intelligence  Analysis:  Does  NSA  Have  What  it  Takes?"  Cryptologic  Quarterly,  Vol. 
20,  No.  1-2,2001. 

225  Moore,  D.T.  &  Krizan,  L.  "Core  Competencies  for  Intelligence  Analysis  at  the  National  Security  Agency,"  in 
Bringing  Intelligence  About:  Practitioners  Reflect  on  Best  Practices,  ed.  R.G.  Swenson,  Joint  Military  Intelligence 
College,  Center  for  Strategic  Intelligence  Research,  Washington,  D.C.,  2003,  p.  96. 
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Figure  10:  Moore  and  Krizan’s  Proposed  Core  Competencies  for  Intelligence  Analysts 

Stephane  Lefebvre  also  highlighted  the  importance  of  critical  thinking  for  intelligence  analysis  in 
2004: 

Critical  thinking  is  particularly  important,  as  analysts  use  human  source  testimony  to 
determine  the  intentions  and  plans  of  an  adversary.  The  intelligence  analyst  must  decide 
into  which  of  mathematician  Pierre  Simon  Laplace’s  four  categories  the  testimonial 
evidence  that  was  collected  falls:  “(1)  the  witness  does  not  deceive  and  is  not  mistaken, 
(2)  the  witness  does  not  deceive  and  is  in  fact  mistaken,  (3)  the  witness  does  deceive  and 
is  not  mistaken,  and  (4)  the  witness  does  deceive  and  is  in  fact  mistaken.”65  Testimony 
that  is  “incomplete,  inconclusive,  and  lacks  credibility  to  some  degree”“must  be 
corroborated  or  graded  (in  terms  of  its  force  or  weight).  To  be  of  any  use,  it  must  have 
credibility  and  probative  force,  and  be  relevant  to  the  requirement  at  hand,  bearing  in 

997  — 

mind  that  experience  and  intuition  may  have  a  role  to  play. 

David  Moore  wrote  Critical  Thinking  and  Intelligence  Analysis  in  2006,  providing  a  thorough 
description  of  critical  thinking  and  how  it  applies  in  intelligence  analysis.  The  book  includes  the 
syllabus  for  a  National  Security  Agency  Critical  Thinking  and  Structured  Analysis  course  that  he 
developed.  Chapter  titles  include: 

•  What  Is  Critical  Thinking? 

o  Defining  Critical  Thinking 

o  Standards  for  Critical  Thinking 

o  Skill-Based  Definitions 

o  A  Disposition  to  Think  Critically 

~27  Lefebvre,  S.  "A  Look  at  Intelligence  Analysis,"  International  Journal  of  Intelligence  and  Counterintelligence, 
Vol.  17,  No.  2,  2004,  p.243. 
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The  Role  of  Questions 
o  Pseudo-Critical  Thinking 

•  What  Can  Be  Learned  from  the  Past? 

•  How  Can  Intelligence  Analysts  Employ  Critical  Thinking? 

•  How  Can  Analysts  be  Taught  to  Think  Critically? 

228 

•  How  Does  Critical  Thinking  Transform?” 

As  the  tenn  critical  thinking  became  more  commonplace,  it  was  used  in  governing  documents 
such  as  the  2006  US  Intelligence  Community’s  Five  Year  Strategic  Human  Capital  Plan.  The 
plan  identified  critical  thinking  as  a  needed  general  analytic  competency.  Note  that  although 
critical  thinking  is  identified,  no  explanation  is  given  to  describe  what  is  meant  by  the  tenn. 

In  partnership  with  the  Deputy  Director  of  National  Intelligence  for  Analysis,  we  have 
completed  the  validation  of  an  IC  analytic  competency  model  that  identifies  the  set  of 
critical  knowledges,  skills,  and  attributes  required  to  conduct  effective  intelligence 
analysis,  now  and  in  the  future.  The  model,  to  be  completed  and  implemented  in  FY 
2006,  includes  general  analytic  competencies  (such  as  critical  thinking);  categories  of 
“target”  expertise,  based  on  the  topics  and  countries  set  forth  in  the  NIPF;  and  analytic 
tradecraft  —  the  tools  and  methods  used  by  the  intelligence  disciplines,  linked  directly  to 

99Q 

IC  quality  standards  for  analytic  products. 

The  Defense  Intelligence  Agency  Strategic  Plan  2007-2012  called  for  improving  critical  thinking 
as  part  of  Goal  3:  Produce  the  right  intelligence  for  the  right  customer  at  the  right  time.  This 
plan  also  introduced  a  new  concept  of  critical  thinking  models,  without  explanation  of  what  is 
meant  by  critical  thinking  models.  Objective  3.3  states: 

Objective  3.3 

Foster  critical  thinking  and  promote  long-term  strategic  analysis  and  warning 
through  the  use  of  advanced  research  methods  and  techniques. 

DIA  will  develop  expertise  in  the  use  of  advanced  analytic  methods  and  techniques  that 
build  on  past  knowledge.  This  includes  the  integration  of  critical  thinking  models  and 
diverse  perspectives  in  the  development  of  analytic  output.  The  Agency  will  provide 
customers  with  analytic  judgments  that  clearly  identify  opportunities,  vulnerabilities,  and 
uncertainties.  DIA  will: 

•  Create  an  environment  for  analysts  to  use  critical  thinking  and  basic  structured 
analysis  techniques  to  mitigate  bias,  understand  mindsets,  and  incorporate 
competing  views  to  help  guide  the  national  dialogue  regarding  threats  to  the  United 
States  and  its  interests. 

•  Institutionalize  forward-looking  strategic  analytic  methods  and  techniques  to 
increase  long-term  strategic  analytic  production  to  ensure  warning  of  future 
threats.230 


228  Moore,  D.T.  Critical  Thinking  and  Intelligence  Analysis,  Joint  Military  Intelligence  College,  Washington,  D.C., 
2006,  pp.  iii-iv. 

229  Anonymous.  The  US  Intelligence  Community's  Strategic  Human  Capital  Plan:  An  Annex  to  the  US  National 
Intelligence  Strategy,  Office  of  the  Director  of  National  Intelligence,  Washington,  D.C.,  2006,  p.  12. 

230  Anonymous.  Strategic  Plan  2007-2012.  Leading  the  Defense  Intelligence  Enterprise,  Defense  Intelligence 
Agency,  Washington,  D.C.,  [undated],  pp.  13-14. 
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As  noted  in  the  sections  of  this  report  on  definitions  and  skills,  ICD  Number  610  explicitly 
identified  critical  thinking  as  a  required  competency  for  the  IC. 

By  2008,  Noel  Hendrickson  can  state  in  Critical  Thinking  in  Intelligence  Analysis,  “Critical 

ry\r) 

thinking  appears  on  almost  every  list  of  the  essential  skills  for  intelligence  analysts.” 

However,  the  enthusiasm  for  critical  thinking  is  not  supported  by  any  demonstration  of  its  ability 
to  improve  the  results  of  intelligence  analysis.  I  found  no  studies  assessing  the  benefit  of  critical 
thinking  to  intelligence  analysis.  Chin  Tam  completed  a  literature  search  of  critical  thinking  and 
intelligence  analysis  in  2009  and  noted  the  same  lack: 

After  the  initial  review  in  current  literature,  we  found  that  few  have  investigated  the  link 
between  critical  thinking  and  intelligence  analysis.  While  there  has  been  work  trying  to 
link  the  two,  there  has  not  been  any  empirical  evidence  showing  the  correlation.  Many 
authors  speculate  that  the  link  does  exist  but  no  definitive  correlation  exists  currently.  In 
order  for  that  to  happen,  two  issues  must  be  resolved:  a)  criteria  matrix  for  scoring 
intelligence  analysis  and  b)  tools  or  methodologies  to  quantitatively  measure  that.  At  the 
present  neither  of  these  issues  has  been  solved.  Within  the  limited  scope  of  this  effort,  it 
may  be  unrealistic  to  establish  a  definitive  link  between  critical  thinking  skills  and 
intelligence  analysis.233 

Within  ten  years  of  its  proposal,  critical  thinking  was  identified  as  a  required  competency. 
However,  this  requirement  was  not  based  on  a  common  understanding  of  critical  thinking  nor  on 
an  evaluation  of  how  it  actually  benefits  intelligence  analysis.  The  next  two  sections  illustrate 
this  concern. 

4.5.2.  What  is  Critical  Thinking  in  Intelligence  Analysis? 

Of  the  106  documents  we  located  that  contained  definitions  of  critical  thinking,  21  documents 
came  from  the  intelligence  analysis  literature.  Seven  of  the  intelligence  documents  referenced 
definitions  from  other  disciplines,  13  of  the  documents  provided  new  definitions,  and  some  did 
both.  Although  common  themes  do  exist  in  the  intelligence  analysis  critical  thinking  definitions, 
some  of  the  definitions  have  unique  aspects,  as  the  samples  in  Table  14  illustrate. 

One  challenge  for  critical  thinking  scholars  working  in  academia  is  that  they  are  attempting  to 
provide  a  generic  definition  that  might  be  applicable  in  many  environments.  A  common 
definition  of  critical  thinking  tailored  to  intelligence  analysis  could  exist  since  the  environment  is 
well-known  to  practitioners  and  scholars  in  the  field. 

Examples  of  the  definitions  I  found  while  searching  the  intelligence  literature  are  shown  in  Table 
14.  The  examples  were  selected  to  illustrate  the  breadth  of  understanding  of  thinking  critically  in 
the  intelligence  literature.  The  source  identifies  whether  the  definition  came  from  another 
discipline  or  was  created  by  the  author  of  the  document.  Some  definitions  were  provided  in 
earlier  sections  of  this  report  and  are  included  again  for  context. 


231  Office  of  the  Director  of  National  Intelligence.  Intelligence  Community’  Directive  Number  610.  Competency 
Directories  for  the  Intelligence  Community’  Workforce.  2008. 

2,2  Hendrickson,  N.  "Critical  Thinking  in  Intelligence  Analysis,"  International  Journal  of  Intelligence  and 
Counterintelligence,  Vol.  21,  No.  4,  2008,  p.  679. 

233  Tam,  C.K.  Behavioral  and  Psychosocial  Considerations  in  Intelligence  Analysis:  A  Preliminary’  Review  of 
Literature  on  Critical  Thinking  Skills,  AFRL-RH-AZ-TR- 1009-0009,  Air  Force  Research  Laboratory,  Mesa,  A Z, 
2009,  p.  2. 
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Table  14:  Critical  Thinking  Definitions  in  the  Intelligence  Analysis  Literature 


Author  and  Source 

Definition 

Moore  and  Krizan 

Source  is  the  Foundation 
for  Critical  Thinking 

An  intellectually  disciplined  process  of  actively  and  skillfully  conceptualizing,  applying,  analyzing, 
synthesizing,  and/or  evaluating  information  gathered  from,  or  generated  by,  observation, 
experience,  reflection,  reasoning,  or  communication,  as  a  guide  to  belief  and  action  ....  Thinking 
about  [our]  thinking  while  [we're]  thinking  in  order  to  make  [our]  thinking  better."’4 

Smith 

Source  is  own  definition 

Critical  thinking  involves  analyzing  a  problem  in  depth,  which  in  turn  often  requires  extrapolation 
and  the  projection  of  observed  patterns  and  trends  into  the  long-  or  short-term  future  based  on 
statistics  and  mathematical  probability."’3 

Waltz 

Source  is  own  definition 

Critical  thinking  is  based  on  the  application  of  a  systematic  method  to  guide  the  collection  of 
evidence,  reason  from  evidence  to  argument,  and  apply  objective  decision-making  judgment  (Table 
4.10).  The  systematic  methodology  assures  completeness  (breadth  of  consideration),  objectivity 
(freedom  from  bias  in  sources,  evidence,  reasoning,  or  judgment),  consistency  (repeatability  over  a 
wide  range  of  problems),  and  rationality  (consistency  with  logic).  In  addition,  critical  thinking 
methodology  requires  the  explicit  articulation  of  the  reasoning  process  to  allow  review  and  critique 
by  others.236 

Bruce  and  George 

Source  is  own  definition, 
with  a  connection  to 
Foundation  for  Critical 
Thinking 

Critical  thinking  is  that  mode  of  thinking  about  any  subject,  content,  or  problem  in  which  the 
individual  improves  the  quality  of  his  or  her  thinking  by  skillfully  analyzing,  assessing,  and 
reconstructing  it.  Critical  thinking  is  largely  self-directed,  self-disciplined,  self-monitored,  and  self¬ 
corrective  thinking.  It  presupposes  rigorous  standards  of  excellence  and  mindful  command  of  their 

237 

use. 

Analysis  101  Participant 
Guide 

Source  is  own  definition 

Critical  thinking  is  quality  control  for  problem  solving  and  reasoning.  Critical  thinking  enables  you 
to  judge  the  accuracy  and  relevance  of  your  information,  categorize  information  as  assumption, 
fact,  or  inference,  identify  gaps  in  your  knowledge,  recognize  factors  that  influence  your 
objectivity  and  the  objectivity  of  others,  and  judge  the  validity  of  your  arguments.238 

Hess  and  Friedel 

Source  is  the  APA 
consensus  definition 

We  understand  critical  thinking  to  be  purposeful,  self-regulatory  judgment  which  results  in 
interpretation,  analysis,  evaluation,  and  inference,  as  well  as  explanation  of  the  evidential, 
conceptual,  methodological,  criteriological,  or  contextual  considerations  upon  which  that  judgment 
is  based.239 

Intelligence  Community 
Directive  Number  610 
Source  is  own  definition 

Critical  Thinking  -  IC  employees  are  expected  to  use  logic,  analysis,  synthesis,  creativity, 
judgment,  and  systematic  approaches  to  gather,  evaluate,  and  use  multiple  sources  of  information 
to  effectively  inform  decisions  and  outcomes. 

•  Creative  Thinking 

•  Exploring  Alternatives 

•  Enterprise  Perspective. 

•  Situational  Awareness 

•  Synthesis 240 

Bennett 

Source  is  own 
definition 

Critical  Thinking.  Uses  logic,  analysis,  synthesis,  creativity,  judgment,  and 
systematic  approaches  to  gather,  evaluate,  and  use  multiple  sources  of 
information  to  inform  decisions  and  outcomes  241 

234  Moore,  D.T.  &  Krizan,  L.  "Intelligence  Analysis:  Does  NSA  Have  What  it  Takes?"  Cryptologic  Quarterly,  Vol. 
20,  No.  1-2,  2001,  p.  14. 

235  Smith,  T.J.  "Predictive  Warning:  Teams,  Networks,  and  Scientific  Method,"  in  Analyzing  Intelligence:  Origins, 
Obstacles,  and  Innovations,  eds.  R.Z.  George  &  J.B.  Bruce,  Georgetown  University  Press,  Washington,  D.C.,  2008, 
p.  270. 

236  Waltz,  E.  Knowledge  Management  in  the  Intelligence  Enterprise,  Artech  House,  Boston,  MA,  2003,  pp.  143-144. 
2  ,7  Bruce,  J.B.  &  George,  R.Z.  "Introduction:  Intelligence  Analysis  -  the  Emergence  of  a  Discipline,"  in  Analyzing 
Intelligence:  Origins,  Obstacles,  and  Innovations,  eds.  R.Z.  George  &  J.B.  Bruce,  Georgetown  University  Press, 
Washington,  D.  C.,  2008,  p.  311. 

2,s  Anonymous.  Analysis  101.  Participant  Guide  2008,  p.l  1. 

239  Hess,  J.  &  Friedel,  C.  "Applying  Critical  Thinking  to  Intelligence  Analysis,"  American  Intelligence  Journal,  Vol. 
26,  No.  1,2008,  p.  31. 

"40  Office  of  the  Director  of  National  Intelligence.  Intelligence  Community  Directive  Number  610.  Competency 
Directories  for  the  Intelligence  Community  Workforce  ,  p.  B-2. 
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Although  the  different  characteristics  found  in  these  definitions  all  appear  valuable,  the  multiple 
definitions  do  not  help  us  understand  unambiguously  what  is  meant  when  the  term  critical 
thinking  is  used  in  an  intelligence  analysis  context.  The  existence  of  these  diverse  definitions 
raises  important  questions: 

•  Which  definition  of  critical  thinking,  if  any,  will  improve  analysis? 

•  Does  critical  thinking  include  creativity  and  an  enterprise  perspective  as  identified 
in  ICD  Number  61 01 

•  Does  it  include  extrapolation  and  projection  as  identified  by  Smith? 

•  Is  it  focused  on  analyzing  and  assessing  your  own  thinking  as  Bruce  and  George 
described  it? 

Sherman  Kent  noted  the  problem  of  ambiguous  tenninology  in  his  1955  essay  “The  Need  for  An 
Intelligence  Literature:” 

If  we  do  not  rigorously  define  our  terms  we  are  likely  to  find  ourselves  talking  at  cross 
purposes;  and  such  discussion,  we  all  realize,  risks  being  more  of  a  fruitless  dispute  than 
an  elevated  debate.242 

It  is  clear  that  the  IC  has  not  yet  reached  the  point  of  having  an  elevated  debate  about  critical 
thinking.  The  confusion  continued  when  authors  attempted  to  describe  how  to  apply  critical 
thinking  to  analysis  work  as  illustrated  in  the  next  section. 

4.5.3.  Applying  Critical  Thinking  to  Intelligence  Analysis 

I  found  very  different  perspectives  for  what  constitutes  thinking  critically  when  performing 
intelligence  analysis.  The  existence  of  these  differing  perspectives  has  a  major  impact  on  how 
critical  thinking  would  be  taught  to  analysts  and  on  the  benefit  that  might  be  realized  from 
thinking  critically.  The  primary  perspectives  are: 

•  Critical  thinking  is  a  quality  of  thinking  that  one  applies  to  every  part  of  the 
analysis  work,  as  concluded  in  this  report  (see  Figure  2). 

•  Critical  thinking  is  the  same  as  or  enabled  by  structured  analysis;  this  approach 

equates  critical  thinking  with  structured  analysis  or  structured  tools.  Promoters 
of  this  perspective  believe  that  if  one  uses  structured  analytic  tools,  one 
automatically  is  thinking  critically. 

•  Generally  confusing  explanations  of  critical  thinking. 

The  same  document  sometimes  contained  different  perspectives.  In  some  cases  the  difference 
existed  because  the  author  of  the  foreword  or  introduction  had  a  different  perspective  than  the 
author  of  the  text.  In  one  document,  however,  the  text  itself  had  conflicting  perspectives. 


~41  Bennett,  B.  "DIA's  Analytic  Competency-Based  Training,"  American  Intelligence  Journal,  Vol.  26,  No.  2, 
2008/2009,  p.  27. 

242  Kent,  S.  "The  Need  for  an  Intelligence  Literature,"  Studies  in  Intelligence,  45th  Anniversary  Special  Edition,  Fall, 

2000,  p.  6. 
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Examples  of  Critical  Thinking  as  a  Quality  of  Thinking :  David  Moore’s  Critical  Thinking  and 
Intelligence  Analysis  described  critical  thinking  as  a  way  to  perform  analysis,  that  is,  a 
qualitative  approach  to  analysis. 

Critical  thinking  provides  that  framework  by  ensuring  that  each  form  of  reasoning  is 
appropriately  used.  Critical  thinking  extends  to  the  entire  intelligence  analysis  process. 
The  claim  here  is  that  analysts  who  become  better  critical  thinkers  will  improve  their 

243 

analyses,  helping  to  lessen  the  likelihood  of  intelligence  failures.” 

Analysis  101,  which  has  also  been  called  Critical  Thinking  and  Structured  Analysis \  described 
critical  thinking  as  a  qualitative  approach  to  conducting  analysis: 

In  this  lesson,  we  lay  the  critical  thinking  and  problem  solving  foundations,  by  offering 
common  definitions,  establishing  a  common  language  and  showing  how  critical  thinking 
can  be  constantly  and  effectively  applied  to  the  art  of  intelligence  analysis.244 

Examples  That  Equate  Critical  Thinking  with  Structured  Analysis  and  Tools:  Promoters  of 
this  perspective  believe  that  if  one  uses  structured  analytic  tools,  one  automatically  is  thinking 
critically.  The  challenge  with  this  perspective  is  that  structured  analysis  consists  of  tools  or 
techniques,  each  of  which  can  be  applied  at  different  points  in  the  analysis  work  to  improve  the 
quality  of  analysis.  Although  critical  thinking  shares  the  intended  outcome  of  improved  analytic 
quality,  an  analyst  can  use  structured  analysis  tools  without  necessarily  engaging  the  critical 
quality  of  their  thinking.  However,  many  in  the  IC  believe  that  critical  thinking  is  the  same  as 
structured  analysis.  For  example,  Michael  W.  Collier  provided  this  description. 

Thanks  to  the  promotion  of  ACH  by  Heuer  and  his  colleagues,  critical  thinking  tools  and 
techniques  are  becoming  more  widely  employed  in  testing  competing  hypotheses  using 
qualitative  data.30  Critical  thinking  skills  that  support  divergent  thinking  include  tools 
such  as  matrix  analyses,  decision  trees,  devil's  advocate  analysis,  weighted  ranking 
analyses,  and  utility  theory.  Such  critical  thinking  skills  should  be  mandatory  tools  in  the 
kit  of  every  intelligence  analyst  -  without  these  tools  the  validity  of  qualitative  analyses 
can  quickly  be  called  into  question.245 

Even  with  Moore’s  clear  description  of  how  critical  thinking  supports  analysis  in  Critical 
Thinking  and  Intelligence  Analysis,  one  of  the  introductions  to  the  book  stated  that  critical 
thinking  is  a  structured  analytic  method.  Jeffrey  Cooper  wrote: 

I  urge  the  leadership  of  the  IC  to  place  far  more  emphasis  on  structured  analytic  methods. 
In  my  view,  the  transformation  of  the  intelligence  enterprise  demands  a  more  curious, 
more  agile,  and  more  deeply  thoughtful  cadre  of  intelligence  analysts  -  but  it  should  also 
require  the  same  traits  among  its  intelligence  organizations  and  the  intelligence  enterprise 
as  a  whole.  Moore  notes  that  "Investment  in  critical  thinking  as  part  of  the  analysis 
process  minimizes  the  likelihood  of  specific  failures"  (page  81).  However,  from  my 
perspective,  critical  thinking  (and  other  structured  methods)  are  more  important  for 


243  Moore,  D.T.  Critical  Thinking  and  Intelligence  Analysis,  Joint  Military  Intelligence  College,  Washington,  D.C., 

2006,  p.  6. 

~44  Anonymous.  Analysis  101.  Participant  Guide  2008,  p.  4. 

245  Collier,  M. W.  "A  Pragmatic  Approach  to  Developing  Intelligence  Analysis,"  Defense  Intelligence  Journal,  Vol. 
14,  No.  2,  2005,  p.  26. 
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changing  the  organization's  overall  approach  to  analysis,  rather  than  in  improving 
specific  judgments  or  preventing  particular  failures.246 

Jack  Davis  wrote  the  introduction  to  Heuer’s  Psychology  of  Intelligence  Analysis.  He  believed 
that  Heuer’s  tools  and  techniques  supported  critical  thinking.  It  should  be  noted  that  Heuer  did 
not  use  the  term  critical  thinking  in  the  book  except  to  state  that  it  is  not  compatible  with  creative 
thinking.  Although  the  concept  of  Structured  Analytic  Techniques  was  not  widely  adopted  at  the 
time  of  this  book,  the  belief  that  using  structured  tools  will  lead  to  critical  thinking  is  clear  in  this 
quote. 

Heuer's  Central  Ideas 

Dick  Heuer's  writings  make  three  fundamental  points  about  the  cognitive  challenges 
intelligence  analysts  face: 

•  The  mind  is  poorly  "wired"  to  deal  effectively  with  both  inherent  uncertainty  (the 
natural  fog  surrounding  complex,  indeterminate  intelligence  issues)  and  induced 
uncertainty  (the  man-made  fog  fabricated  by  denial  and  deception  operations). 

•  Even  increased  awareness  of  cognitive  and  other  "unmotivated"  biases,  such  as  the 
tendency  to  see  information  confirming  an  already-held  judgment  more  vividly  than 
one  sees  "disconfirming"  information,  does  little  by  itself  to  help  analysts  deal 
effectively  with  uncertainty. 

•  Tools  and  techniques  that  gear  the  analyst's  mind  to  apply  higher  levels  of  critical 
thinking  can  substantially  improve  analysis  on  complex  issues  on  which 
information  is  incomplete,  ambiguous,  and  often  deliberately  distorted.  Key 
examples  of  such  intellectual  devices  include  techniques  for  structuring 

'y  An 

information,  challenging  assumptions,  and  exploring  alternative  interpretations.' 

Examples  of  Confusing  Explanations  of  Critical  Thinking:  Richards  Heuer  described  critical 
thinking  as  an  activity  rather  than  as  a  quality  of  thinking  in  Taxonomy  of  Structured  Analytic 
Techniques.  Interestingly  enough,  he  also  explicitly  excluded  it  from  structured  analysis.  Heuer 
identified  four  general  categories  of  analysis  methods:  quantitative  methods  using  empirical  data; 

248 

quantitative  methods  using  expert-generated  data;  unaided  judgment;  and  structured  analysis.' 

He  described  critical  thinking  as  useful  only  in  unaided  judgment: 

•  Unaided  Judgment ::  This  is  how  most  strategic  and  political  analysis  is  now  being 
done.  It  includes  evidentiary  reasoning,  the  basics  of  critical  thinking,  historical 
method,  case  study  method,  and  reasoning  by  analogy.3  One  very  distinctive 
characteristic  of  unaided  judgment  is  that  it  is  usually  an  individual  effort  in  which 
the  thinking  remains  largely  in  the  mind  of  the  individual  analyst  until  it  is  written 
down  in  a  draft  report.  Training  in  this  type  of  analysis  is  generally  provided 
through  graduate  education,  especially  in  the  social  sciences  and  liberal  arts. 

•  Structured  Analysis:  This  uses  structured  techniques  to  mitigate  the  adverse  impact 
on  our  analysis  of  known  cognitive  limitations  and  pitfalls.  The  most  distinctive 


746  Moore,  p.  xiv. 

247  Heuer,  R.J.J.  Psychology’  of  Intelligence  Analysis,  Center  for  the  Study  of  Intelligence,  Washington,  D.C.,  1999, 
p.  10. 

~48  Heuer,  R.J.  "Taxonomy  of  Structured  Analytic  Techniques,"  International  Studies  Association  2008  Annual 
Convention,  San  Francisco,  CA,  March  26  -  29,  2008,  pp.  2-3. 
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characteristic  is  that  structured  techniques  externalize  and  decompose  our  thinking 
in  a  manner  that  enables  it  to  be  reviewed  and  critiqued  piece  by  piece,  or  step  by 
step,  by  other  knowledgeable  analysts.  These  techniques  can  be  used  by  the  average 
analyst  who  lacks  advanced  training  in  statistics,  math,  or  the  hard  sciences.  For 
most  analysts,  training  in  structured  analytic  techniques  is  obtained  only  within  the 
Intelligence  Community.249 

Structured  Analytic  Techniques  for  Intelligence  Analysis  by  Richards  Heuer  and  Randolph 
Pherson  continued  the  confusion  about  the  relationship  between  critical  thinking  and  structured 
analysis.  As  with  Taxonomy  of  Structured  Analytic  Techniques  referenced  above,  Heuer  and 
Pherson  identified  critical  thinking  as  useful  in  the  expert  judgment  methodology  (earlier  called 
unaided  judgment)  rather  than  as  part  of  the  structured  analysis  methodology.  The  text  in  both 
books  is  similar.  However,  in  the  final  chapter  of  Structured  Analytic  Techniques  for 
Intelligence  Analysis,  Heuer  and  Pherson  stated  that  critical  thinking  is  a  structured  technique. 

The  process  for  coordinating  a  National  Intelligence  Estimate  (NIE)  has  changed 
dramatically.  Formal  coordination  prior  to  publication  of  an  NIE  is  now  usually  a 
formality,  as  there  has  been  collaboration  among  interested  parties  from  the  inception, 
and  all  relevant  intelligence  is  shared.  The  Key  Assumptions  Check  and  other  basic 
critical  thinking  and  structured  analytic  techniques  have  identified  and  explored 
differences  of  opinion  early  in  the  preparation  of  an  estimate,  and  new  analysis 
techniques,  such  as  Premortem  Analysis,  Structured  Self-Critique,  and  Adversarial 
Collaboration  -  all  now  described  and  discussed  in  the  DNI  knowledge  management 
system  -  are  being  employed  to  define  and  resolve  disagreements  as  much  as  possible 
prior  to  the  final  coordination.250 

Wayne  Michael  Hall  and  Gary  Citrenbaum  identified  critical  thinking  as  one  of  14  elements  of 
advanced  analysis  in  their  book,  Intelligence  Analysis:  How  to  Think  in  Complex  Environments. 

It  is  not  at  all  clear  from  the  text  what  the  authors  understand  critical  thinking  to  be.  Hall  and 
Citrenbaum  used  the  following  definition  for  critical  thinking,  which  they  note  came  from  David 
G.  Myers  in  Exploring  Psychology. 

An  intellectual  process  that  "examines  assumptions,  discerns  hidden  values,  evaluates 

25 1 

evidence,  and  assesses  conclusions." 

Their  explanation  of  critical  thinking  is  less  helpful.  The  chapter  on  critical  thinking  addresses 
examining  assumptions  from  the  definition,  but  then  reviews  several  other  topics  including  the 
role  of  “deep  think”  in  critical  thinking,  thinking  errors,  collection  errors,  and  thinking  like  the 
enemy.  The  chapter  ends  with  this  summary: 

Critical  thinking  is  important  for  bringing  forth  the  best  in  thinking,  because  it  asks  the 
analyst  to  think  in  three  spheres: 

First,  the  analyst  must  internalize  critical  thinking  and  consider  how  he  or  she  is  thinking 
as  an  individual. 


24 '  Heuer,  pp.  2-3. 

250  Heuer,  R.J.  &  Pherson,  R.H.  Structured  Analytic  Techniques  for  Intelligence  Analysis,  CQ  Press,  Washington, 
D.C.,  2011,  p.  342. 

251  Hall,  W.M.  &  Citrenbaum,  G.M.  Intelligence  Analysis:  How  to  Think  in  Complex  Environments,  Praeger  Security 
International,  Santa  Barbara,  CA,  2010,  p.  31. 
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Second,  the  analyst  must  seek  help  from  outsiders  to  help  with  thinking,  as  is  the  case 
with  indigenous  people  and  technical  experts,  and  to  examine  his  or  her  thoughts 
critically. 

Third,  the  analyst  must  be  critical  of  the  way  other  people  are  thinking  and  learn  to  spot 
errors.  They  are  obliged  to  challenge  what  appears  to  be  erroneous  assumptions  and  poor 
thinking.252 

These  conflicting  perspectives  about  how  critical  thinking  supports  analysis  illustrate  the  need 
for  the  intelligence  community  to  reach  a  common  understanding  of  all  aspects  of  critical 
thinking  within  analysis.  We  must  agree  whether  critical  thinking  is  a  quality  of  thinking  used  in 
all  aspects  of  intelligence  analysis,  a  structured  tool  similar  to  ACH,  or  a  simple  activity  that  is 
used  occasionally. 

This  is  an  area  needing  additional  attention  in  order  to  gain  the  maximum  benefit  from 
investments  in  developing  critical  thinking  competencies. 

4.5.4.  Summary:  Critical  Thinking  in  the  IC  Literature 

Critical  thinking  could  be  a  useful  descriptor  for  the  cognitive  aspects  of  analysis  if  a  shared 
understanding  existed  of  what  is  meant  by  the  term.  However,  the  literature  actually  contains 
competing  descriptions  of  what  critical  thinking  is  and  how  it  should  be  applied  to  benefit 
intelligence  analysis. 

The  IC  has  not  explored  the  enablers  of  critical  thinking:  thinking  skills,  thinking  dispositions, 
and  personal  epistemology.  Although  only  seven  documents  discussed  thinking  skills,  those 
documents  presented  different  perspectives  of  the  necessary  skills.  Only  one  document 
addressed  dispositions  needed  to  think  critically,  and  no  documents  addressed  personal 
epistemology  as  a  foundation  for  effective  thinking. 

We  need  to  have  the  elevated  debate  called  for  by  Sherman  Kent  to  eliminate  the  ambiguity  and 
confusion  surrounding  the  term  critical  thinking.  The  IC  needs: 

•  A  shared  understanding  of  what  critical  thinking  is 

•  Agreement  about  the  skills,  dispositions,  and  personal  epistemology  needed  to 
support  intelligence  analysis 

•  Agreement  about  how  critical  thinking  should  be  applied  to  the  analysis  process 


252  Hall  &  Cetrenbaum,  p.  120. 
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5.0  CONCLUSIONS 

5.1  Findings 

The  answers  to  the  basic  questions  guiding  this  research  are: 

1 .  What  is  critical  thinking?  We  do  not  have  a  clear  answer. 

a.  Although  critical  thinking  is  generally  recognized  as  important  (and  scholars 
in  all  disciplines  in  this  study  make  this  case  emphatically),  a  formal 
agreement  on  what  critical  thinking  is  does  not  exist  either  within  the 
intelligence  community  or  within  other  disciplines.  However,  common 
themes  do  exist  that  clarify  critical  thinking  and  identify  its  potential 
usefulness  in  intelligence  analysis. 

b.  Critical  thinking  is  a  concept  describing  a  quality  of  thinking.  It  refers  to 
how  one  approaches  the  thinking  task.  Critical  thinking  is  useful  when 
exploring  a  situation,  phenomenon,  question,  or  problem  to  arrive  at  a 
hypothesis  or  conclusion  about  it  that  integrates  all  available  information 
and  that  therefore  can  be  convincingly  justified. 

c.  Skills  and  dispositions  such  as  those  listed  in  Table  15  are  hows  that  might 
be  especially  useful  in  contributing  to  the  quality  of  thinking  called  critical 
thinking.  As  a  quality  of  thought,  critical  thinking  can  and  potentially 
should  be  applied  to  every  part  of  intelligence  analysis. 

Table  15:  Recommended  Critical  Thinking  Skills  and  Dispositions  for  Intelligence  Analysts 


Critical  Thinking  Skills 

Critical  Thinking  Dispositions 

Interpretation 

Willingness  to  engage  in  intellectual 

Analysis 

effort 

Evaluation 

Systematic 

Inference 

Persistent 

Synthesis 

Intellectual  integrity 

Explanation 

Open-minded 

Self-regulation 

Intellectual  curiosity 

2.  What  is  the  state  of  knowledge  of  critical  thinking?  The  best  description  of  the 

current  state  of  knowledge  was  provided  by  Condon  and  Riley:  “The  current 

253 

literature  on  critical  thinking  is  rife  with  conflict  and  competing  ideologies.”" 

a.  Examples  supporting  this  statement  include  the  many  different  definitions, 
lists  of  skills  and  kinds  of  skills,  and  models  of  critical  thinking  that  I  found 
in  the  literature. 

b.  Little  attention  has  been  devoted  to  critical  thinking  performance  -  are 
individuals  with  strong  critical  thinking  skills  and  dispositions  as  measured 
by  assessments  able  to  perform  better  in  work  that  appears  to  require  critical 
thinking?  This  shortcoming  is  as  apparent  in  the  intelligence  literature  as  it 
is  in  the  literature  from  other  disciplines. 


253  Condon,  W.  &  Kelly-Riley,  D.  "Assessing  and  Teaching  What  We  Value:  The  Relationship  between  College- 
Level  Writing  and  Critical  Thinking  Abilities,"  Assessing  Writing,  Vol.  9,  No.  1,  2004,  p.  64. 
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3. 


Can  we  train  someone  to  think  critically?  Perhaps. 

a.  Much  of  the  education  and  training  available  to  date  in  university  courses, 
publicly  available  courses,  and  courses  intended  specifically  for  the 
intelligence  community  are  based  on  simplistic  descriptions  of  critical 
thinking,  provide  insufficient  time  for  effective  practice,  use  ineffective 
teaching  strategies,  and  include  few  or  ineffective  methods  of  assessing 
what  has  been  learned. 

b.  Training  that  has  been  effective  as  demonstrated  by  perfonnance  on  some 
sort  of  assessment  was  developed  to  teach  a  specific  set  of  skills, 
dispositions,  and  operations  in  order  to  achieve  a  specific  outcome. 
Successful  training  has  been  focused,  well-designed,  and  consistently 
delivered,  with  extensive  opportunities  for  practice. 

4.  Can  we  assess  someone ’s  ability  to  think  critically?  Again,  perhaps. 

a.  A  number  of  standardized  assessments  and  examples  of  customized 
assessments  exist  that  address  different  aspects  of  critical  thinking.  It  is  not 
clear  whether  any  of  the  currently  available  standardized  assessments  are 
beneficial  for  the  IC.  Few  documents  were  found  that  described 
assessments  designed  for  adults  to  be  used  in  determining  job  readiness, 
suitability,  or  performance.  '  No  documents  were  found  describing  how  to 
assess  whether  someone  uses  critical  thinking  regularly  in  his  or  her  job  or 
life. 

b.  Each  publicly  available  assessment  is  somewhat  different,  focused  on  the 
critical  thinking  definition  espoused  by  the  assessment  author. 

c.  Publicly  available  assessments  have  primarily  been  used  in  assessing  the 
impact  that  a  class  or  an  experience  such  as  years  in  college  has  on  the 
ability  to  think  critically.  That  is,  they  assess  an  experience’s  impact  at 
Level  2  in  Breckenridge’s  schema.255 

d.  Several  documents  provided  guidelines  for  developing  a  custom  assessment 
(i.e.  rubrics)  that  could  be  designed  for  a  unique  environment  such  as  the 
intelligence  community.  Those  publications  also  provided  examples  of 
custom  assessment  tools. 

5.  How  can  this  understanding  of  critical  thinking  be  applied  to  improving  intelligence 

analysis?  The  intelligence  community  needs  to  exert  additional  effort  in  order  to 

benefit  from  investments  in  critical  thinking. 

a.  Critical  thinking  is  widely  recognized  as  an  important  competency  for 
intelligence  analysts.  However,  the  IC  does  not  yet  have  a  common 
understanding  of  what  critical  thinking  is  or  how  to  apply  it  in  intelligence 
analysis  work.  Without  a  common  understanding  of  critical  thinking,  the 
claims  regarding  its  ability  to  improve  intelligence  analysis,  the  investments 
in  training  for  it,  and  documents  such  as  strategic  plans  calling  for  critical 
thinking  are  built  on  a  weak  foundation.  Which  version  of  critical  thinking  is 


254  One  exception  is  a  study  of  using  the  Watson  Glaser  Critical  Thinking  Assessment  in  hiring  intelligence  analysts 
completed  in  1983. 

255  Breckenridge,  J.G.  "Designing  Effective  Teaching  and  Learning  Environments  for  a  New  Generation  of 
Analysts,"  International  Journal  of  Intelligence  and  Counterintelligence,  Vol.  23,  No.  2,  2010,  p.  308. 
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expected  to  improve  analysis?  What  critical  thinking  competencies  will 
result  in  better  analysis?  How  would  critical  thinking  be  recognized  if  it 
were  present? 

b.  Current  training  in  the  IC  varies  widely.  Courses  exist  with  different 
definitions  of  critical  thinking,  a  variety  of  topics  covered,  and  varying 
amounts  of  practice  available,  resulting  in  analysts  who  have  differing  levels 
of  critical  thinking  knowledge  and  abilities.  Many  courses  are  primarily 
designed  to  introduce  analysts  to  the  concept  of  critical  thinking.  These 
courses  do  not  provide  sufficient  depth  or  time  for  practice  that  would 
enable  an  analyst  to  become  a  skilled  critical  thinker.  I  found  no  reports  of 
training  evaluations  that  demonstrated  that  the  courses  actually  improved 
analyst  competencies  or  perfonnance. 

c.  Although  it  is  widely  believed  that  critical  thinking  will  improve 
intelligence  analysis,  I  found  no  reports  demonstrating  the  benefit  of  critical 
thinking  to  intelligence  analysis. 

5.2  Opportunities  for  the  Future 

These  findings  provide  a  roadmap  for  research  and  activities  that  will  demonstrate  the  utility  of 
critical  thinking  to  the  IC.  These  activities  will  also  identify  the  best  way  to  integrate  critical 
thinking  into  intelligence  analysis  to  gain  maximum  benefit  throughout  the  entire  IC. 

1 .  Define  a  concept  of  critical  thinking  that  makes  sense  for  intelligence  analysis  and 
that  can  be  accepted  throughout  the  IC.  This  definition  should: 

a.  Identify  expected  standards  of  critical  thinking  in  intelligence  analysis  such 
as  was  presented  with  the  sample  rubrics  discussed  in  the  assessment  section 
of  this  report. 

b.  Identify  the  appropriate  skills,  dispositions,  and  personal  epistemology 
maturity  level  needed  for  effective  intelligence  analysis. 

c.  Clearly  articulate  how  to  apply  critical  thinking  in  intelligence  analysis. 

2.  Evaluate  the  various  aspects  of  critical  thinking  prior  to  making  major  investments 
in  fostering  it  throughout  the  IC.  Research  projects  might  include: 

a.  Investigating  the  impact  that  improved  critical  thinking  skills  and 
dispositions  have  on  intelligence  analysis. 

b.  Determining  an  approach  to  training  (including  topics,  teaching  strategies, 
and  practice  mechanisms)  that  will  provide  the  most  improvement  to  the 
practitioners  of  intelligence  analysis. 

c.  Evaluating  publicly  available  assessments  and  their  usefulness  for  assessing 
an  analyst’s  skills  or  dispositions  and  therefore  predicting  job  performance 
or  identifying  training  needs. 

d.  Evaluating  whether  assessments  of  critical  thinking  skills,  dispositions,  and 
personal  epistemology  are  useful  in  the  hiring  process. 

3.  Develop  a  consistent  critical  thinking  training  curriculum  that  includes  topics  that 
have  been  demonstrated  to  improve  intelligence  analysis  and  that  uses  best  practices 
in  skills  training.  Rigorously  evaluate  the  effectiveness  of  the  training  and  make 
appropriate  improvements  until  the  training  meets  the  needs  of  the  IC. 
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a.  The  successful  experiments  described  in  section  4.3.2.  of  this  report  provide 
models  of  effective  instruction. 

b.  The  training  used  in  nursing  education  provides  a  cautionary  tale  of  training 
approaches  to  avoid. 

Detennine  the  best  way  to  assess  the  thinking  skills  and  dispositions  of  analysts  and 
their  ability  to  apply  critical  thinking  to  the  analysis  process.  Develop  an 
assessment  methodology  that  measures  not  only  the  results  of  training,  but  the 
impact  of  that  training  on  analytic  results. 

a.  For  example,  the  sample  rubrics  included  in  the  assessment  section  of  this 
report  provide  information  that  is  not  only  useful  for  assessment,  but  also 
useful  in  identifying  perfonnance  expectations  in  advance.  An  analyst  could 
use  a  rubric  to  guide  his  or  her  work  efforts  and  evaluate  his  or  her  own 
analytic  products. 
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APPENDIX  A  -  Critical  Thinking  Definitions 


Appendix  A  contains  all  definitions  found  during  the  search  conducted  for  this  research  effort.  Source  indicates  the  original  source  of  the  definition  such  as 
the  Foundation  for  Critical  Thinking  or  the  APA  study  convened  by  Facione. 


Definition 

Author/Field 

Date 

Source 

1.  Thinking  that  helps  students  understand  the  logical  connectives  of  English  (Adler) 

Lippman 

Education 

2003 

Adler 

2.  Peter  Facione  convened  a  group  of  individuals  considered  experts  in  the  field  of  critical 
thinking  to,  among  other  tasks,  develop  a  consensus  definition  of  critical  thinking: 

EXPERT  CONSENSUS  STATEMENT  REGARDING  CRITICAL  THINKING  AND  THE 
IDEAL  CRITICAL  THINKER  “We  understand  critical  thinking  to  be  purposeful,  self- 
regulatory  judgment  which  results  in  interpretation,  analysis,  evaluation,  and  inference,  as 
well  as  explanation  of  the  evidential,  conceptual,  methodological,  criteriological,  or 
contextual  considerations  upon  which  that  judgment  is  based.  CT  is  essential  as  a  tool  of 
inquiry.  As  such,  CT  is  a  liberating  force  in  education  and  a  powerful  resource  in  one’s 
personal  and  civic  life.  While  not  synonymous  with  good  thinking,  CT  is  a  pervasive  and 
self-rectifying  human  phenomenon.  The  ideal  critical  thinker  is  habitually  inquisitive,  well- 
informed,  trustful  of  reason,  openminded,  flexible,  fair-minded  in  evaluation,  honest  in 
facing  personal  biases,  prudent  in  making  judgments,  willing  to  reconsider,  clear  about 
issues,  orderly  in  complex  matters,  diligent  in  seeking  relevant  information,  reasonable  in 
the  selection  of  criteria,  focused  in  inquiry,  and  persistent  in  seeking  results  which  are  as 
precise  as  the  subject  and  the  circumstances  of  inquiry  permit.  Thus,  educating  good  critical 
thinkers  means  working  toward  this  ideal.  It  combines  developing  CT  skills  with  nurturing 
those  dispositions  which  consistently  yield  useful  insights  and  which  are  the  basis  of  a 
rational  and  democratic  society.” 

Facione 

Philosophy 

1990 

APA 

3.  We  understand  critical  thinking  to  be  purposeful,  self-regulatory  Judgment  which  results  in 
Interpretation,  analysis,  evaluation,  and  inference,  as  well  as  explanation  of  the  evidential, 
conceptual,  methodological,  criteriological,  or  contextual  considerations  upon  which  that 
judgment  is  based...  CT  is  essential  as  a  tool  of  inquiry.  As  such,  CT  is  a  liberating  force  in 
education  and  a  powerful  resource  in  one's  personal  and  civic  life...  While  not  synonymous 
with  good  thinking,  CT  is  a  pervasive  and  self-rectifying  human  phenomenon. 

Facione  et  al. 
Philosophy 

1994 

APA 
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4.  An  outcome  from  Facione's  study  was  a  classification  of  important  skills  and  sub-skills 
which  included  interpretation,  analysis,  evaluation,  inference,  explanation,  and  self¬ 
regulation.  This  classification  system  was  the  basis  of  the  Delphi  study  that  we  conducted 
with  faculty,  employers,  and  policymakers. 

Jones  et  al. 
Education 

1995 

APA 

5.  critical  thinking  is  “purposeful,  self-regulatory  judgment  which  results  in  interpretation, 
analysis,  evaluation,  and  inference  as  well  as  explanation  of  the  evidential,  conceptual, 
methodological,  criteriological,  or  contextual  considerations  upon  which  that  judgment  is 
based.” 

Spicer  &  Flanks 
Communications 

1995 

APA 

6.  The  Delphi  experts  defined  critical  thinking  as  “purposeful,  self-regulatory  judgment  which 
results  in  interpretation,  analysis,  evaluation,  and  inference,  as  well  as  explanation  of  the 
evidential,  conceptual,  methodological,  criteriological,  or  contextual  considerations  upon 
which  that  judgment  is  based.” 

Reed 

Education 

1998 

APA 

7.  We  understand  critical  thinking  to  be  purposeful,  self-regulatory  judgment  which  results  in 
interpretation,  analysis,  evaluation,  and  inference,  as  well  as  explanation  of  the  evidential, 
conceptual,  methodological,  criteriological,  or  contextual  considerations  upon  which  that 
judgment  is  based. 

Facione 

Philosophy 

2000 

APA 

8.  We  understand  critical  thinking  to  be  purposeful,  self-regulatory  judgment  which  results  in 
interpretation,  analysis,  evaluation,  and  inference,  as  well  as  explanation  of  the  evidential, 
conceptual,  methodological,  criteriological,  or  contextual  considerations  upon  which  that 
judgment  is  based.  CT  is  essential  as  a  tool  of  inquiry.  As  such,  CT  is  a  liberating  force  in 
education  and  a  powerful  resource  in  one's  personal  and  civic  life. 

Montgomery 

Medical 

2003 

APA 

9.  Facione  asserts  that  critical  thinking  is  “purposeful,  self-regulatory  judgment  which  results 
in  interpretation,  analysis,  evaluation,  and  inference  as  well  as  explanation  of  the  evidential, 
conceptual,  methodological,  criteriological,  or  contextual  considerations  upon  which  that 
judgment  is  based.” 

Kelly-Riley  et  al. 
Education 

2004 

APA 

10.  We  understand  critical  thinking  to  be  purposeful,  self-regulatory  judgment  which  results  in 
interpretation,  analysis,  evaluation,  and  inference,  as  well  as  explanation  of  the  evidential, 
conceptual,  methodological,  criteriological,  or  contextual  considerations  upon  which  that 
judgment  is  based.  CT  is  essential  as  a  tool  of  inquiry. 

Hess  &  Friedel 

1A 

2008 

APA 
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1 1 .  We  understand  critical  thinking  to  be  purposeful,  self-regulatory  judgment  which  results  in 
interpretation,  analysis,  evaluation,  and  inference,  as  well  as  explanation  of  the  evidential, 
conceptual,  methodological,  criteriological,  or  contextual  considerations  upon  which  that 
judgment  is  based.  (2009  update  to  the  1990  report  found  in  3634) 

Facione 

Philosophy 

2009 

APA 

12.  Becoming  conscious  of  one's  own  thinking  so  as  to  be  able  to  transfer  it  from  familiar  to 
unfamiliar  contexts  (Arons) 

Lippman 

Education 

2003 

Arons 

13.  In  my  analysis,  epistemological  understanding  is  not  simply  a  sub-component.  Rather  it  is 
the  central  concept  through  which  critical  thinking  is  conceptualized.  It  is  that  which 
undeipins,  justifies,  and  makes  sense  of  the  activities  and  dispositions  related  to  reason- 
assessment. 

Bailin 

Education 

1997 

Bailin 

14.  Critical  thinking  is  the  assessing  of  the  authenticity,  accuracy  and/or  worth  of  knowledge 
claims  and  arguments.  (Beyer's  own  definition) 

Beyer 

Education 

1985 

Beyer 

15.  Critical  thinking  is  unique  because  it  involves  careful  precise  persistent  and  objective 
analysis  of  any  knowledge  claim  or  belief  to  judge  its  validity  and/or  worth. 

Beyer 

Education 

1985 

Beyer 

16.  SC  task  force  developed  a  definition  that  they  adapted  from  several  folks:  Critical  thinking 
is  a  reflective,  systematic,  rational,  and  skeptical  use  of  cognitive  representations,  processes, 
and  strategies  about  beliefs,  problems,  and/or  courses  of  action. 

Cook  et  al. 
Education 

1996 

Beyer,  Ennis, 
Facione, 
McPeck, 
Sternberg 

17.  'Critical  thinking  involves  calling  into  question  the  assumptions  underlying  our  customary, 
habitual  ways  of  thinking  and  acting  and  then  being  ready  to  think  and  act  differently  on  the 
basis  of  this  critical  questioning.'  (Brookfield) 

Hare 

Education 

1999 

Brookfield 

18.  The  shorthand  for  critical  thinking  that  has  become  most  popular,  probably  because  of  an 
exercise  we  do,  is  "thinking  outside  of  the  box." 

Eichhom 

Army 

2010 

Creative 

thinking 

19.  Critical  thinking  consists  essentially  of  "evaluating  statements,  arguments  and  experiences." 
(D'Angelo,  1971) 

Beyer 

Education 

1985 

D’Angelo 

20.  In  its  most  aggressive  form  "the  spotting  of  faults."  (DeBono,  1983) 

Beyer 

Education 

1985 

DeBono 

21.  Active,  persistent,  and  careful  consideration  of  any  belief  or  supposed  form  of  knowledge  in 
the  light  of  the  grounds  that  support  it  and  the  further  conclusions  to  which  it  tends 
constitutes  reflective  thought.  Any  one  of  the  first  three  kinds  of  thought  may  elicit  this 
type;  but  once  begun,  it  includes  a  conscious  and  voluntary  effort  to  establish  belief  upon  a 
firm  basis  of  evidence  and  rationality. 

Dewey 

Education, 

philosophy, 

psychology 

1910, 

1934 

Dewey 
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22.  Critical  thinking  calls  for  a  persistent  effort  to  examine  any  belief  or  supposed  form  of 
knowledge  in  the  light  of  the  evidence  that  supports  it  and  the  further  conclusions  to  which 
it  tends.  (Glaser,  1941) 

Although  Beyer  credits  Glaser,  the  original  source  is  Dewey 

Beyer 

Education 

1985 

Dewey 

23.  Reflective  thinking:  Reflection  is  the  process  of  critically  assessing  the  content,  process,  or 
premise(s)  of  our  efforts  to  inteipret  and  give  meaning  to  an  experience,  (p.  104) 

Also  notes  that  “Much  of  the  current  confusion  in  academic  circles  about  the  nature  of  critical 
thinking  is  the  result  of  failing  to  differentiate  among  the  three  functions  of  reflection  and  to 
distinguish  between  reflective  and  nonreflective  action.”  (p.  106) 

Mezirow 

Education 

1991 

Dewey 

24.  Active,  persistent,  and  careful  consideration  of  a  belief  or  supposed  form  of  knowledge  in 
the  light  of  the  grounds  which  support  it  and  the  further  conclusions  to  which  it  tends. 

Fisher 

Philosophy 

2001 

Dewey 

25.  A  model  more  generally  applied  across  disciplines,  critical  thinking,  developed  out  of  the 
Socratic  tradition  of  disciplined  inquiry.  Usually  defined  as  a  reasoned  or  questioning 
approach  in  which  one  examines  assumptions  and  seeks  evidence  (Donald,  1985), 
researchers  suggest  that  critical  thinking  includes  components  of  logic,  problem  solving,  and 
Piagetian  formal  operations  (Meyers,  1986;  Sternberg,  1985). 

Hativa  & 
Marincovich, 
Education 

1995 

Donald 

26.  Critical  thinking  is  reasonable  reflective  thinking  that  is  focused  on  deciding  what  to  believe 
or  do. 

Ennis 

Education 

1987 

Ennis 

27.  Critical  thinking  is  reasonable  reflective  thinking  that  is  focused  on  deciding  what  to  believe 
or  do. 

Ennis 

Education 

1989 

Ennis 

28.  The  conception  of  critical  thinking  that  is  spelled  out  in  this  analysis  is  one  we  share.  It 
involves  the  active  and  appropriate  use  of  key  critical-thinking  skills  in  judging  the 
reasonableness  of  ideas  and  the  justification  of  actions,  set  in  the  context  of  attitudes  and 
dispositions  that  place  value  on  careful  and  open  thinking.  In  critical  thinking,  so  construed, 
we  search  for  good  reasons  that  aid  us  in  making  decisions. 

Norris  &  Ennis 
Education 

1989 

Ennis 

29.  We  feel  that  the  following  basic  definition  comes  closer  to  expressing  the  way  the  term  is 
generally  used  in  education:  Critical  thinking  is  reasonable  and  reflective  thinking  that  is 
focused  upon  deciding  what  to  believe  or  do. 

Norris  &  Ennis 
Education 

1989 

Ennis 

30.  Another  way  to  describe  it  is  reasonable,  reflective,  responsible,  and  skillful  thinking  that  is 
focused  on  deciding  what  to  believe  or  do. 

Schafersman 

Education 

1991 

Ennis 

31.  Thus  broadly  conceived,  CT  was  characterized  as  purposeful,  self-regulatory  judgment,  a 
human  cognitive  process.  As  a  result  of  this  non-linear,  recursive  process  a  person  forms  a 
judgment  about  what  to  believe  or  what  to  do  In  a  given  context 

Facione  et  al. 
Education 

1994 

Ennis 
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32.  More  recently,  Ennis  (1987,  p.  10)  defined  critical  thinking  as  "reasonable,  reflective 
thinking  that  is  focused  on  deciding  what  to  believe  or  do." 

King  &  Kitchener 
Educational 
psychology 

1994 

Ennis 

33.  This  is  a  doc  bv  Jones  for  office  of  educational  research.  Cited  multiple  definitions.  Robert 
Ennis  ( 1987)  defines  critical  thinking  as  "reasonable  reflective  thinking  that  is  focused  on 
deciding  what  to  believe  or  do" 

Jones  et  al. 
Education 

1995 

Ennis 

34.  reasonable  reflective  thinking  that  is  focused  on  deciding  what  to  believe  or  do 

Bailin 

Education 

1997 

Ennis 

35.  The  definition  of  critical  thinking  currently  cited  by  those  concerned  with  critical  reading  is 
the  one  advanced  by  Ennis:  “reasonable,  reflective  thinking  that  is  focused  on  deciding  what 
to  believe  and  do”  (Norris  &  Ennis,  1989,  p.  1).  The  transformation  of  critical  reading  and 
thinking  is  revealed  by  comparing  this  definition  to  the  earlier  definition  advanced  by  Ennis 
in  1962:  “the  correct  assessing  of  statements”  (p.  81).  The  newer  definition  attends  to  the 
reflective  nature  of  critical  thinking,  holding  that  thinkers  must  consciously  seek  and  use 
good  reasons,  and  it  places  deciding  what  to  believe  and  do  as  the  goal  of  critical  thinking. 

Douglas 

Education 

2000 

Ennis 

36.  R:  The  general  consensus  is  that  CT  per  se  is  judging  in  a  reflective  way  what  to  do  or  what 
to  believe.  The  cognitive  skills  of  analysis,  interpretation,  inference,  explanation, 
evaluation,  and  of  monitoring  and  correcting  one’s  own  reasoning  are  at  the  heart  of  critical 
thinking. 

Facione 

Philosophy 

2000 

Ennis 

37.  CT  is  judgment,  reflective  and  purposive. 

Facione 

Philosophy 

2000 

Ennis 

38.  Thus  conceived,  CT  was  characterized  as  a  self-adjusting  process  of  judging  what  to  believe 
or  what  to  do  in  a  given  context,  (pp.  7-8) 

Facione 

Philosophy 

2000 

Ennis 

39.  Reasonable,  reflective  thinking  about  what  to  do  and  believe 

Hatcher 

Education/philosophy 

2000 

Ennis 

40.  From  Ennis:  reflective  thinking  focused  on  deciding  what  to  believe  or  do 

Jonassen 

Education 

2000 

Ennis 

41.  Critical  thinking  is  reasonable,  reflective  thinking  that  is  focused  on  deciding  what  to 
believe  or  do.  (Cf.  Norris  and  Ennis,  1989) 

Fisher 

Philosophy 

2001 

Ennis 

42.  Reasonable  reflective  thinking  that  is  focused  on  deciding  what  to  believe  or  do  (Ennis) 

Lippman 

Education 

2003 

Ennis 

43.  Critical  thinking  is  reasonable  reflective  thinking  focused  on  what  to  believe  or  do"  (Ennis, 
1993). 

Montgomery 

Medical 

2003 

Ennis 
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44.  new  CT  definitions  specifically  for  nursing: 

Kataoka- Yahiro  and  Saylor  (1994)  defined  critical  thinking  as  reflective  and  reasoned 
thinking  about  nursing  problems  with  more  than  one  solution,  focusing  on  decisions  about 
what  to  believe  and  do. 

Staib 

Nursing 

2003 

Ennis 

45.  The  Ennis  definition:  Critical  thinking  is  reasonable  and  reflective  thinking  focused  on 
deciding  what  to  believe  or  do. 

Ennis 

Education 

2005 

Ennis 

46.  The  following  definition  seems  to  be  more  in  accord  with  contemporary  usage  and  thus.  I 
hope,  will  minimize  confusion  in  communication:  "Critical  thinking  is  reasonable  reflective 
thinking  focused  on  deciding  what  to  believe  or  do.'" 

Ennis 

Education 

1993 

Ennis 

47.  In  our  view,  thinking  critically  involves  the  ongoing  judgment  of  one's  own  thinking,  a  view 
which  is  consistent  with  the  spirit  of  Ennis'  current  definition:  "Critical  thinking  is 
reasonable  reflective  thinking  that  is  focused  on  deciding  what  to  believe  and  do" 

Tierney  et  al. 
Education 

1989 

Ennis 

48.  For  this  paper  critical  thinking  is  the  propensity  and  skills  to  engage  in  an  activity  with 
reflective  skepticism  focused  on  deciding  what  to  reasonably  believe  or  do.  Three  broad 
criteria  can  be  extracted  from  this  definition:  thinking  must  be  reasonable,  reflective,  and 
focused  to  be  considered  critical  thinking.  Critical  thinkers  must  not  only  possess  the  set  of 
skills  we  have  labeled  the  “reason  assessment”  component  (elementary  clarification,  basic 
support,  inference,  advanced  clarification,  and  strategies),  but,  also  have  the  “critical  spirit” 
component. 

McKown 

Military 

1997 

Ennis  and 
McPeck 

49.  Critical  thinking  involves  thinking  through  problematic  situations  about  what  to  believe  or 
how  to  act  where  the  thinker  makes  reasoned  judgments  that  embody  the  attributes  of 
quality  thinking.  Uncritical  thinking:  thinking  that  accepts  conclusions  at  face  value  without 
any  assessment  of  their  merits  or  bases 

Bailin 

Education 

1993 

Ennis  derivative 

50.  6)  It  involves  in  its  most  benign  form  "  the  correct  assessing  of  statements"  (Ennis,  1962) 

Beyer 

Education 

1985 

Ennis  original 

51.  Beyer  uses  the  earlier  version  from  Ennis:  Experts  in  the  study  of  critical  thinking  have  for 
years  been  rather  specific  about  what  they  mean  by  the  term.  Critical  thinking,  according  to 
them  and  as  used  here,  means  judging  the  authenticity,  worth,  or  accuracy  of  something. 

Beyer 

Education 

1987 

Ennis  original 

52.  As  a  root  notion,  critical  thinking  is  taken  to  be  the  correct  assessing  of  statements.  Since 
there  are  various  kinds  of  statements,  various  relations  between  statements  and  their 
grounds,  and  various  stages  in  the  process  of  assessment,  we  can  expect  that  there  will  be 
various  ways  of  going  wrong  when  one  attempts  to  think  critically. 

King  &  Kitchener 
Educational 
psychology 

1994 

Ennis  original 
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53.  The  correct  assessing  of  statements  (Ennis) 

Lippman 

Education 

2003 

Ennis  original 

54.  Critical  thinking  is  the  correct  assessment  of  statements 

Ennis 

Education 

1963 

Ennis  original 

55.  the  correct  assessing  of  statements. 

Bailin 

Education 

1997 

Ennis  original 

56.  correct  assessing  of  statements 

Ennis 

Education 

1963 

Ennis  original 

57.  Correct  assessing  of  statements 

Ennis 

Education 

1962 

Ennis  original 

58.  A  definition  of  critical  thinking  that  I  at  one  time  endorsed  is  that  critical  thinking  is  the 
correct  assessing  of  statements  (Ennis,  1962). 

Ennis 

Education/philosophy 

1993 

Ennis  original 

59.  Critical  thinking  is  a  nonlinear,  recursive  process  in  which  a  person  forms  a  judgment  about 
what  to  believe  or  what  to  do  in  a  given  context" 

Credited  to  Facione,  but  is  not  the  APA  def. 

Myrick 

Nursing 

1998 

Facione,  related 
to  Ennis 

60.  Critical  thinking  is  the  judging  of  statements  based  on  acceptable  standard.  (Feeley  1976) 

Beyer 

Education 

1985 

Feeley,  related 
to  Ennis 

61.  Critical  thinking  is  skilled  and  active  interpretation  and  evaluation  of  observations  and 
communications,  information  and  argumentation. 

Fisher 

Philosophy 

2001 

Fisher 

62.  Critical  thinking  is  skilled,  active  interpretation  and  evaluation  of  observations, 

communications,  information,  and  argumentation.  (Note:  they  also  mention  that  you  can 
add  ...as  a  guide  to  thought  and  action,  to  the  definition,  but  they  do  not  see  the  need  to  do 
so.) 

Fisher  &  Scriven 
Philosophy 

1997 

Fisher  & 
Scriven 

63.  what  I  call  strong  sense  critical  thinking:  (a)  an  ability  to  question  deeply  his  own 

framework  of  thought,  (b)  an  ability  to  reconstruct  sympathetically  and  imaginatively  the 
strongest  versions  of  points  of  view  and  frameworks  of  thought  opposed  to  his  own,  and  (c) 
an  ability  to  reason  dialectically  (multilogically)  in  such  a  way  as  to  determine  when  his 
own  point  of  view  is  at  its  weakest  and  when  an  opposing  point  of  view  is  at  its  strongest. 

(p.  377) 

Paul 

Philosophy 

1987 

Foundation 
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64.  Critical  thinking  according  to  Richard  Paul  (19931  is  "disciplined,  self-directed  thinking 
which  exemplifies  the  perfections  of  thinking  appropriate  to  a  particular  mode  or  domain  of 
thinking"  (p.  462).  It  is  an  art  of  thinking  about  one's  thinking  in  order  to  make  it  more  clear, 
accurate,  or  "more  defensible."  This  type  of  thinking  requires  "the  mastery  of  intellectual 
skills  and  abilities." 

Jones  et  al. 
Education 

1995 

Foundation 

65.  R.  Paul  -  thinking  about  your  thinking  while  you’re  thinking  to  make  your  thinking  better 

Reed 

Education 

1998 

Foundation 

66.  R.  Paul  -  a  unique  kind  of  purposeful  thinking  in  which  the  thinker  systematically  and 

habitually  imposes  criteria  and  intellectual  standards  upon  the  thinking,  taking  charge  of  the 
construction  of  thinking,  guiding  the  construction  of  the  thinking  according  to  the  standards, 
assessing  the  effectiveness  of  the  thinking  according  to  the  purpose,  the  criteria,  and  the 
standards. 

Reed 

Education 

1998 

Foundation 

67.  Richard  Paul  and  Michael  Scriven  have  defined  critical  thinking  as  "the  intellectual 
disciplined  process  of  actively  and  skillfully  conceptualizing,  applying,  analyzing, 
synthesizing,  and/or  evaluating  information  gathered  from,  or  generated  by,  observations, 
experience,  reflection,  reasoning,  or  communication  as  a  guide  to  belief  and  action" 

Hatcher 

Education/philosophy 

2000 

Foundation 

68.  Critical  thinking  is  that  mode  of  thinking  -  about  any  subject,  content  or  problem  -  in  which 
the  thinker  improves  the  quality  of  his  or  her  thinking  by  skillfully  taking  charge  of  the 
structures  inherent  in  thinking  and  imposing  intellectual  standards  upon  them.  (Paul,  Fisher 
and  Nosich,1993,  p.  4) 

Fisher 

Philosophy 

2001 

Foundation 

69.  An  intellectually  disciplined  process  of  actively  and  skillfully  conceptualizing,  applying, 
analyzing,  synthesizing,  and/or  evaluating  information  gathered  from,  or  generated  by, 
observation,  experience,  reflection,  reasoning,  or  communication,  as  a  guide  to  belief  and 
action  ....  Thinking  about  [our]  thinking  while  [we're]  thinking  in  order  to  make  [our] 
thinking  better. 

Moore 

IA 

2001 

Foundation 

70.  Thinking  that  aims  to  overcome  bias,  prejudice,  and  stereotyping  (Paul) 

Lippman 

Education 

2003 

Foundation 

71.  Thinking  that  aims  to  protect  us  from  deceptions  by  others  and  from  self-deception  (Paul) 

Lippman 

Education 

2003 

Foundation 

72.  Thinking  about  thinking 

Lippman 

Education 

2003 

Foundation 

73.  The  ability  of  thinkers  to  take  charge,  to  develop  intellectual  standards  and  apply  them  to 
their  own  thinking  (Paul) 

Lippman 

Education 

2003 

Foundation 
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74.  An  intellectually  disciplined  process  of  actively  and  skillfully  conceptualizing,  applying, 
analyzing,  synthesizing,  and/or  evaluating  information  gathered  from,  or  generated  by, 
observation,  experience,  reflection,  reasoning,  or  communication,  as  a  guide  to  belief  and 
action  ....  Thinking  about  [our]  thinking  while  [we're]  thinking  in  order  to  make  [our] 
thinking  better. 

Moore 

IA 

2003 

Foundation 

75.  An  intellectually  disciplined  process  of  actively  and  skillfully  conceptualizing,  applying, 
analyzing,  synthesizing,  and/or  evaluating  information  gathered  from,  or  generated  by, 
observation,  experience,  reflection,  reasoning,  or  communication,  as  a  guide  to  belief  and 
action  ....  Thinking  about  [our]  thinking  while  [we're]  thinking  in  order  to  make  [our] 
thinking  better. 

Moore  &  Krizan 

IA 

2003 

Foundation 

76.  From  Paul:  “a  unique  kind  of  purposeful  thinking  in  which  the  thinker  systematically  and 
habitually  imposes  criteria  and  intellectual  standards  upon  the  thinking.... 

Cohen  et  al. 

Army 

2004 

Foundation 

77.  Paul,  Elder,  and  Bartell  (1997)  defined  critical  thinking  in  their  study  on  faculty  knowledge 
as  “thinking  that  explicitly  aims  at  well-founded  judgment  and  hence  utilizes  appropriate 
evaluative  standards  in  the  attempt  to  determine  the  true  worth,  merit,  or  value  of 
something.” 

Kelly-Riley  et  al. 
Education 

2004 

Foundation 

78.  Scriven  and  Paul  (2003)  define  critical  thinking  as  “the  intellectually  disciplined  process  of 
actively  and  skillfully  conceptualizing,  applying,  analyzing,  synthesizing,  and/or  evaluating 
information  gathered  from,  or  generated  by  observation,  experience,  reflection,  reasoning,  or 
communication  as  a  rubric  to  belief  and  action.” 

Kelly-Riley  et  al. 
Education 

2004 

Foundation 

79.  Uses  one  of  Richard  Paul's  definitions:  Critical  Thinking:  (1)  Disciplined,  self-directed 
thinking  that  exemplifies  the  perfections  of  thinking  appropriate  to  a  specific  mode  or 
domain  of  thinking;  (2)  thinking  that  displays  mastery  of  intellectual  skills  and  abilities;  (3) 
the  art  of  thinking  about  one’s  thinking  while  thinking,  to  make  one’s  thinking  better:  more 
clear,  more  accurate,  or  more  defensible;  (4)  thinking  that  is  fully  aware  of  and  continually 
guards  against  the  natural  human  tendency  to  self-deceive  and  rationalize  to  selfishly  get 
what  it  wants 

Guillot 

Military 

2004 

Foundation 

80.  Skillfully  taking  charge  of  the  structures  inherent  in  thinking  and  imposing  intellectual 
standards  upon  them. 

McLaughlin 

IA 

2005 

Foundation 
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81.  One  way  to  summarize  the  essence  of  critical  thinking  is  as  follows:  Critical  thinking  is  the 
art  of  thinking  about  thinking  in  such  a  way  as  to: 

1 .  identify  its  strengths  and  weaknesses,  and 

2.  recast  it  in  improved  form  (where  necessary). 

The  first  characteristic  requires  the  thinker  to  be  skilled  in  analytic  and  evaluative  thinking. 
The  second  requires  the  thinker  to  be  skilled  in  creative  thinking.  Thus,  critical  thinking  has 
three  dimensions:  the  analytic,  the  evaluative,  and  the  creative. 

Paul  &  Elder 
Education/philosophy 

2005 

Foundation 

82.  the  ability  to  step  out  of  one's  thoughts,  if  you  will,  to  examine  them  and  the  process  that 
brought  them  about  even  while  you  are  thinking  them. 

Moore 

IA 

2006 

Foundation 

83.  Critical  Thinking  is  a  deliberate  meta-cognitive  (thinking  about  thinking)  and  cognitive 
(thinking)  act  whereby  a  person  reflects  on  the  quality  of  the  reasoning  process 
simultaneously  while  reasoning  to  a  conclusion.  The  thinker  has  two  equally  important 
goals:  coming  to  a  solution  and  improving  the  way  she  or  he  reasons. 

Moore 

IA 

2006 

Foundation 

84.  Critical  thinking  is  the  process  of  analyzing  and  assessing  thinking  with  a  view  to  improving 
it.  Critical  thinking  presupposes  knowledge  of  the  most  basic  structures  in  thinking  (the 
elements  of  thought)  and  the  most  basic  intellectual  standards  for  thinking  (universal 
intellectual  standards).  The  key  to  the  creative  side  of  critical  thinking  (the  actual  improving 
of  thought)  is  in  restructuring  thinking  as  a  result  of  analyzing  and  effectively  assessing  it. 

Paul  &  Elder 
Education/philosophy 

2006 

Foundation 

85.  Critical  thinking  is  the  art  of  analyzing  and  evaluating  thinking  with  a  view  to  improving  it. 

Paul  &  Elder 
Education/philosophy 

2007 

Foundation 

86.  "Critical  thinking  is  that  mode  of  thinking-about  any  subject,  content,  or  problem-in  which 
the  thinker  improves  the  quality  of  his  or  her  thinking  by  skillfully  analyzing,  assessing,  and 
reconstructing  it.  Critical  thinking  is  self-directed,  self-disciplined,  self-monitored,  and  self¬ 
corrective  thinking' 

Analysis  101 

IA 

2008 

Foundation 

87.  Critical  thinking  is  that  mode  of  thinking  about  any  subject,  content,  or  problem  in  which 
the  individual  improves  the  quality  of  his  or  her  thinking  by  skillfully  analyzing,  assessing, 
and  reconstructing  it.  Critical  thinking  is  largely  self-directed,  self-disciplined,  self- 
monitored,  and  self-corrective  thinking.  It  presupposes  rigorous  standards  of  excellence  and 
mindful  command  of  their  use. 

Bruce  &  George 

IA 

2008 

Foundation 

88.  ...the  intellectually  disciplined  process  of  actively  and  skillfully  conceptualizing,  applying, 
analyzing,  synthesizing,  and/or  evaluating  information  gathered  from,  or  generated  by, 
observation,  experience,  reflection,  reasoning,  or  communication  as  a  guide  to  belief  and 
action.  (Scriven,  1996) 

Krupat 

Medical 

2008 

Foundation 
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89.  . .  .the  art  of  thinking  about  your  thinking  while  you  are  thinking  in  order  to  make  your 
thinking  better:  more  clear,  more  accurate,  and  more  defensible.  (Paul  et  al.,  1989)  (p.  5)  : 

Krupat 

Medical 

2008 

Foundation 

90.  Critical  thinking  is  self-guided,  self-disciplined  thinking  which  attempts  to  reason  at  the 
highest  level  of  quality  in  a  fair-minded  way.  Linda  Elder 

The  page  also  included  a  link  to  a  "more  complete"  definition  of  CT. 

Elder 

Education 

2009 

Foundation 

91.  Critical  thinking  is  the  intellectually  disciplined  process  of  actively  and  skillfully 

conceptualizing,  applying,  analyzing,  synthesizing,  and/or  evaluating  information  gathered 
from,  or  generated  by,  observation,  experience,  reflection,  reasoning,  or  communication,  as 
a  guide  to  belief  and  action.  In  its  exemplary  form,  it  is  based  on  universal  intellectual 
values  that  transcend  subject  matter  divisions:  clarity,  accuracy,  precision,  consistency, 
relevance,  sound  evidence,  good  reasons,  depth,  breadth,  and  fairness..." 

Paul  &  Scriven 
Philosophy 

2009 

Foundation 

92.  "Critical  thinking  at  AMSC  is  defined  as  disciplined,  self-directed  thinking  displaying  a 
mastery  of  intellectual  skills  and  abilities — thinking  about  your  thinking  while  you’re 
thinking  to  make  your  thinking  better." 

Eichhom 

Army 

2010 

Foundation 

93.  We  also  developed  a  series  of  shorthand  labels  such  as  "thinking  about  thinking" 

Eichhom 

Army 

2010 

Foundation 

94.  Critical  thinking  is  the  intellectually  disciplined  process  of  actively  and  skillfully 

conceptualizing,  applying,  analyzing,  synthesizing,  and/or  evaluating  information  gathered 
from,  or  generated  by,  observation,  experience,  reflection,  reasoning,  or  communication,  as 
a  guide  to  belief  and  action.  In  its  exemplary  form,  it  is  based  on  universal  intellectual 
values  that  transcend  subject  matter  divisions:  clarity,  accuracy,  precision,  consistency, 
relevance,  sound  evidence,  good  reasons,  depth,  breadth,  and  fairness. 

Paul  &  Scriven 
Philosophy 

undated 

Foundation 

95.  Critical  thinking  is  self-guided,  self-disciplined  thinking  which  attempts  to  reason  at  the 
highest  level  of  quality  in  a  fair-minded  way. 

Paul  &  Scriven 
Philosophy 

undated 

Foundation 

96.  Critical  thinking  is  that  mode  of  thinking  -  about  any  subject,  content,  or  problem  -  in  which 
the  thinker  improves  the  quality  of  his  or  her  thinking  by  skillfully  taking  charge  of  the 
structures  inherent  in  thinking  and  imposing  intellectual  standards  upon  them. 

Paul  &  Scriven 
Philosophy 

undated 

Foundation 

97.  Critical  thinking  is,  in  short,  self-directed,  self-disciplined,  self-monitored,  and  self¬ 
corrective  thinking. 

Paul  &  Scriven 
Philosophy 

undated 

Foundation 

98.  Several  definitions  cited  in  course  materials,  Analysis  101.  First  was  "The  art  of  thinking 
about  your  thinking  while  you  are  thinking  in  order  to  make  your  thinking  better:  more 
clear,  more  accurate,  or  more  defensible" 

Analysis  101 

IA 

2008 

Foundation, 
Binker, 
Adamson,  & 
Martin 
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99.  As  the  former  it  involves  (1)  an  alertness  to  the  need  to  evaluate  information,  (2)  a 
willingness  to  test  opinions  and  (3)  a  desire  to  consider  all  viewpoints. 

Beyer 

Education 

1985 

Fraser  &  West 

100.  Attentiveness  to  the  formal  aspects  of  thinking  (Garver) 

Lippman 

Education 

2003 

Garver 

101.  Discussion  of  argumentative  literature  drawn  from  traditional  works  of  the  humanities 

Lippman 

Education 

2003 

Garver  and 
Adler 

102.  The  ability  to  think  critically,  as  conceived  in  this  volume,  involves  three  things:  (1)  an 
attitude  of  being  disposed  to  consider  in  a  thoughtful  way  the  problems  and  subjects  that 
come  within  the  range  of  one's  experiences,  (2)  knowledge  of  the  methods  of  logical  inquiry 
and  reasoning,  and  (3)  some  skill  in  applying  those  methods.  Critical  thinking  calls  for  a 
persistent  effort  to  examine  any  belief  or  supposed  form  of  knowledge  in  the  light  of  the 
evidence  that  supports  it  and  the  further  conclusions  to  which  it  tends.  It  also  generally 
requires  ability  to  recognize  problems,  to  find  workable  means  for  meeting  those  problems, 
to  gather  and  marshal  pertinent  information,  to  recognize  unstated  assumptions  and  values, 
to  comprehend  and  use  language  with  accuracy,  clarity,  and  discrimination,  to  interpret 
data,  to  appraise  evidence  and  evaluate  arguments,  to  recognize  the  existence  (or  non¬ 
existence)  of  logical  relationships  between  propositions,  to  draw  warranted  conclusions  and 
generalizations,  to  put  to  test  the  conclusions  and  generalizations  at  which  one  arrives,  to 
reconstruct  one's  patterns  of  beliefs  on  the  basis  of  wider  experience,  and  to  render  accurate 
judgments  about  specific  things  and  qualities  in  everyday  life. 

Glaser 

Education 

1941 

Glaser 

With  references 
to  Dewey 

103.  An  attitude  of  being  disposed  to  consider  in  a  thoughtful,  perceptive  manner  the 

problems  and  subjects  that  come  within  the  range  of  one's  experience.  2.  Knowledge  of  the 
methods  of  logical  inquiry  and  reasoning.  3.  Skill  in  applying  those  methods. 

Zechmeister  & 
Johnson 
Philosophy 

1992 

Glaser 

104.  Watson  and  Glaser  (1964)  view  critical  thinking  as  a  composite  of  three  elements:  "(1) 
attitudes  of  inquiry  that  involve  an  ability  to  recognize  the  existence  of  problems  and  an 
acceptance  of  the  general  need  for  evidence  in  support  of  what  is  asserted  to  be  true;  (2) 
knowledge  of  the  nature  of  valid  inferences,  abstractions,  and  generalizations  in  which  the 
weight  or  accuracy  of  different  kinds  of  evidence  are  logically  determined;  and  (3)  skills  in 
employing  and  applying  the  above  attitudes  and  knowledge" 

King  &  Kitchener 
Educational 
psychology 

1994 

Glaser 
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105.  (1)  an  attitude  of  being  disposed  to  consider  in  a  thoughtful  way  the  problems  and 

subjects  that  come  within  the  range  of  one's  experience;  (2)  knowledge  of  the  methods  of 
logical  enquiry  and  reasoning;  and  (3)  some  skill  in  applying  those  methods.  Critical 
thinking  calls  for  a  persistent  effort  to  examine  any  belief  or  supposed  form  of  knowledge  in 
the  light  of  the  evidence  that  supports  it  and  the  further  conclusions  to  which  it  tends. 

(Glaser,  1941,  p.  5) 

Fisher 

Philosophy 

2001 

Glaser  With 
references  to 
Dewey 

106.  Halpem's  (1984)  book  on  critical  thinking  makes  no  mention  of  reflection  and  defines 
critical  thinking  merely  as  "thinking  that  is  purposeful  and  goal  directed"  (p.  3). 

Mezirow 

Education 

1991 

Halpern 

107.  Halpern  -  thinking  that  is  purposeful,  reasoned,  and  goal  directed.  It  is  the  kind  of 

thinking  involved  in  solving  problems,  formulating  inferences,  calculating  likelihoods,  and 
making  decisions 

Reed 

Education 

1998 

Halpern 

108.  Halpern  (1997)  asserts  that  critical  thinking  is  the  “use  of  those  cognitive  skills  or 

strategies  that  increase  the  probability  of  a  desirable  outcome.  It  is  used  to  describe  thinking 
that  is  purposeful,  reasoned  and  goal  directed”  (p.  4).  From  a  cognitive  psychologist’s  view, 
she  cites  several  other  definitions  from  that  perspective:  critical  thinking  is  the  “formation  of 
logical  inferences”;  it  is  the  development  of  cohesive  and  logical  reasoning  and  patterns;  it 
is  careful  and  deliberate  determination  of  whether  to  accept,  reject  or  suspend  judgment;  it  is 
mental  activity  useful  for  a  particular  cognitive  task  (1997,  p.  4). 

Kelly-Riley  et  al. 
Education 

2004 

Halpern 

109.  I  will  use  Diane  Halpem’s  broad  definition  of  critical  thinking  as  a  foundation:  “Critical 
thinking  is  the  use  of  those  cognitive  skills  or  strategies  that  increase  the  probability  of  a 
desirable  outcome.  It  is  used  to  describe  thinking  that  is  purposeful,  reasoned,  and  goal 
directed.”5  In  essence,  critical  thinking  is  about  improving  one’s  judgment. 

Gerras 

Army 

2006 

Halpern 

110.  For  example,  Diane  Halpern  considers  that  [critical]  thinking  is  the  use  of  those 
cognitive  skills  or  strategies  that  increase  the  probability  of  a  desirable  outcome.  It  is... 
thinking  that  is  purposeful,  reasoned,  and  goal  directed  -  the  kind  of  thinking  involved  in 
solving  problems,  formulating  inferences,  calculating  likelihoods,  and  making  decisions, 
when  the  thinker  is  using  skills  that  are  thoughtful  and  effective  to  the  particular  context  and 
type  of  thinking  task. 

Moore 

IA 

2006 

Halpern 

111.  Critical  thinking  is  the  use  of  those  cognitive  skills  or  strategies  that  increase  the 

probability  of  a  desirable  outcome.  It  is  used  to  describe  thinking  that  is  purposeful, 
reasoned,  and  goal  directed. 

Allen  &  Gerras 
Army 

2009 

Halpern 
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112.  Halpem's  recommended  definition:  Critical  thinking  is  the  use  of  those  cognitive  skills 
or  strategies  that  increase  the  probability  of  a  desirable  outcome.  It  is  used  to  describe 
thinking  that  is  purposeful,  reasoned,  and  goal  directed — the  kind  of  thinking  involved  in 
solving  problems,  formulating  inferences,  calculating  likelihoods,  and  making  decisions, 
when  the  thinker  is  using  skills  that  are  thoughtful  and  effective  for  the  particular  context 
and  type  of  thinking  task. 

Halpem 

Psychology 

2003 

Halpem 

113.  Critical  thinking  is  thinking  that  attempts  to  arrive  at  a  judgment  only  after  honestly 

evaluating  alternatives  with  respect  to  available  evidence  and  arguments 

Hatcher 

Education/philosophy 

2000 

Hatcher 

114.  the  honest  evaluation  of  alternatives  with  respect  to  available  evidence  and  arguments. 

Hatcher 

Education 

2001 

Hatcher 

115.  Thinking  that  attempts  to  arrive  at  a  judgment  only  after  honestly  evaluating  alternatives 
with  respect  to  available  evidence  and  arguments  (Hatcher) 

Lippman 

Education 

2003 

Hatcher 

116.  IC  employees  are  expected  to  use  logic,  analysis,  synthesis,  creativity,  judgment,  and 
systematic  approaches  to  gather,  evaluate,  and  use  multiple  sources  of  information  to 
effectively  inform  decisions  and  outcomes. 

Subskills  include:  creative  thinking,  exploring  alternatives,  enterprise  perspective,  situational 
awareness,  synthesis 

Spracher 

IA 

2009 

Intel 

Community 
Directive  610 

117.  From  the  NDIC  Catalog.  Includes  the  following  topics  as  part  of  critical  thinking: 
Situational  awareness  (global/area),  creative  thinking,  synthesis,  exploring  alternatives 

NDIC 

IA 

2008 

Intel 

Community 
Directive  610 

Competencies  for  supervisory  and  managerial  IC  employees  at  GS-15  and  below: 

118.  IC  employees  are  expected  to  use  logic,  analysis,  synthesis,  creativity,  judgment,  and 
systematic  approaches  to  gather,  evaluate,  and  use  multiple  sources  of  information  to 
effectively  inform  decisions  and  outcomes.  In  addition,  IC  supervisors  are  expected  to 
establish  a  work  environment  where  employees  feel  free  to  engage  in  open,  candid 
exchanges  of  information  and  diverse  points  of  view. 

Subskills  include:  Decisiveness,  Flexibility,  Problem  Solving 

Office  of  the  Director 
of  National 
Intelligence 

IA 

2008 

Intel 

Community 
Directive  610 

Competencies  for  non-supervisory  IC  employees  at  GS-15  and  below: 

119.  IC  employees  are  expected  to  use  logic,  analysis,  synthesis,  creativity,  judgment,  and 
systematic  approaches  to  gather,  evaluate,  and  use  multiple  sources  of  information  to 
effectively  inform  decisions  and  outcomes. 

Subskills  include:  creative  thinking,  exploring  alternatives,  enterprise  perspective,  situational 
awareness,  synthesis 

Office  of  the  Director 
of  National 
Intelligence 

IA 

2008 

Intel 

Community 
Directive  610 
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120.  Critical  Thinking.  Uses  logic,  analysis,  synthesis,  creativity,  judgment,  and  systematic 
approaches  to  gather,  evaluate,  and  use  multiple  sources  of  information  to  inform  decisions 
and  outcomes 

Bennett 

IA 

2008-09 

Intel 

Community 
Directive  610 

121.  Critical  thinking  involves  the  dynamic  reorganization  of  knowledge  in  meaningful  and 
usable  ways.  It  involves  three  general  skills:  evaluating,  analyzing,  and  connecting 

Jonassen 

Education 

2000 

Iowa  Dept  of 
Educ 

122.  Jones  and  his  colleagues  (Jones,  Dougherty,  Fantaske,  &  Hoffman,  1995;  Jones, 

Hoffman,  Moore,  Ratcliff,  Tibbetts,  &  Click,  1995)  obtained  consensus  from  among  500 
policymakers,  employers,  and  educators,  who  agree  that  critical  thinking  is  a  broad  term  that 
describes  reasoning  in  an  open-ended  manner  and  with  an  unlimited  number  of  solutions.  It 
involves  constructing  a  situation  and  supporting  the  reasoning  that  went  into  a  conclusion. 

Halpem 

Psychology 

2003 

Jones 

123.  Critical  thinking  is  "a  process  wherein  an  interaction  occurs  between  individuals  and  the 
interpretations  of  knowledge  which  they  create"  (Jones  &  Brown,  1991) 

Perciful  &  Nester 
Nursing 

1996 

Jones  &  Brown 

124.  Knox  (1977),  who  reviews  over  1,000  studies  of  adult  development  and  learning,  only 
once  mentions  critical  thinking,  which  he  characterizes  as  including  "the  ability  to  interpret 
data,  weigh  evidence,  and  engage  in  deductive  thinking"  (p.  446). 

Mezirow 

Education 

1991 

Knox 

125.  Thinking  as  argument  (Kuhn,  1991,  1992;  Yeh,  2002)  offers  a  way  to  define  what  we 
mean  by  critical  thinking,  (p.  172) 

Kuhn 

Educational 

psychology 

2005 

Kuhn 

126.  a  conceptualization  of  critical  thinking  as  careful  argumentation,  a  notion  rooted  in 

classical  rhetoric  and  elaborated  by  Billig  and  Kuhn  (Billig,  1987;  Kuhn,  1991,  1992,  1999) 

Yeh 

Education 

2001 

Kuhn,  Billig 
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127.  Critical  thinking  is  a  rational  response  to  questions  that  cannot  be  answered  definitively 
and  for  which  all  the  relevant  information  may  not  be  available.  It  is  defined  here  as  an 
investigation  whose  purpose  is  to  explore  a  situation,  phenomenon,  question,  or  problem 
to  arrive  at  a  hypothesis  or  conclusion  about  it  that  integrates  all  available  information 
and  that  can  therefore  be  convincingly  justified.  In  critical  thinking,  all  assumptions  are 
open  to  question,  divergent  views  are  aggressively  sought,  and  the  inquiry  is  not  biased  in 
favor  of  a  particular  outcome. 

The  outcomes  of  a  critical  inquiry  are  twofold:  a  conclusion  (or  hypothesis)  and  the 
justification  offered  in  support  of  it.  These  outcomes  are  usually  set  forth  in  the  form  of  an 
argument,  defined  as  "the  sequence  of  interlinked  claims  and  reasons  that,  between  them, 
establish  the  content  and  force  of  the  position  for  which  a  particular  speaker  is  arguing" 
(Toulmin,  Rieke,  and  Janik  1979,  p.  13).  The  need  for  justification  arises  from  the  ill- 
defined  nature  of  problems  to  which  the  term  "critical  thinking"  generally  applies.  Because 
conclusions  cannot  be  tested  (as  they  can  be  in  problem  solving),  the  arguer  must 
demonstrate  their  plausibility  by  offering  supporting  reasons  (Voss,  Tyler,  and  Yengo 

1983). 

The  inquiry  itself,  in  which  evidence  is  reviewed  and  interpreted,  is  sometimes  referred  to  as 
"the  context  of  discovery" ;  it  is  the  inventive,  creative  phase  of  critical  thinking.  The 
presentation  of  the  argument  is  referred  to  as  "the  context  of  justification  "  (Kahane  1980; 
McPeck  1981).  In  practice,  the  two  may  be  intertwined  rather  than  distinct,  (p.  20) 

Kurfiss 

Education 

1988 

Kurfiss 

128.  an  investigation  whose  purpose  is  to  explore  a  situation,  phenomenon,  question,  or 

problem  to  arrive  at  a  hypothesis  or  conclusion  about  it  that  integrates  all  available 
information  and  that  therefore  can  be  convincingly  justified. 

Inch  &  Wamick 
Communications 

2010 

Kurfiss 

129.  Definitions  of  critical  thinking,  such  as  that  of  Lipman  (1988,  p.  39),  who  argues  that 
"critical  thinking  is  skillful,  responsible  thinking  that  facilitates  good  judgment  because  it 
(1)  relies  upon  criteria,  (2)  is  selfcorrecting,  and  (3)  is  sensitive  to  context." 

King  &  Kitchener 
Educational 
psychology 

1994 

Lipman 

130.  Matt  Lipman  defines  critical  thinking  as  "skillful,  responsible  thinking  that  facilitates 
good  judgment  because  it  1)  relies  on  criteria,  2)  is  self-correcting  and  3)  is  sensitive  to 
context" 

Hatcher 

Education/philosophy 

2000 

Lipman 

131.  'Critical  thinking  is  thinking  that  facilitates  judgment  because  it  relies  on  criteria,  is  self- 
correcting,  and  is  sensitive  to  context.'  (Lipman) 

Hare 

Education 

1999 

Lipman 
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132.  Litecky  (1992)  defines  critical  thinking  as  "the  active,  mental  effort  to  make  meaning  of 
our  world  by  carefully  examining  thought  in  order  to  better  understand  content"  (p.  83). 

Jonassen 

Education 

2000 

Litecky 

133.  Thinking  aimed  at  the  integration  of  thought  and  action  (J.  R.  Martin) 

Lippman 

Education 

2003 

Martin 

134.  it  is  the  appropriate  use  of  reflective  scepticism  within  the  problem  area  under 
consideration. 

McPeck 

Educational 

philosophy 

1981 

McPeck 

135.  Critical  thinking  has  two  important  dimensions.  It  is  both  a  frame  of  mind  and  a  number 
of  specific  mental  operations  (McPeck,  1981) 

Beyer 

Education 

1985 

McPeck 

136.  McPeck  (1990)  views  critical  thinking  as  important  within  the  context  of  the  disciplines 
and  the  knowledge  within  those  fields.  Only  through  an  immersion  in  the  disciplines  can 
individuals  fully  develop  their  abilities  to  think  critically.  McPeck  includes  actions  as  well 
as  beliefs  in  his  definition.  He  also  advocates  the  importance  of  "reflective  skepticism" 
which  encourages  the  individual  to  find  the  faults  of  others. 

Jones  et  al. 
Education 

1995 

McPeck 

137.  John  McPeck  defines  critical  thinking  as  "the  skill  and  propensity  to  engage  in  an 
activity  with  reflective  skepticism" 

Hatcher 

Education/philosophy 

2000 

McPeck 

138.  Thinking  that  enables  critical  skills  to  be  transferred  to  educational  subjects 

Lippman 

Education 

2003 

McPeck 

139.  Thinking  that  comes  in  or  may  come  in  when  we  suspect  something  to  be  amiss 

Lippman 

Education 

2003 

McPeck 

140.  Thinking  that  represents  the  philosophy  of  x,  for  the  philosophy  of  x  should  ideally  be 
an  integral  part  of  what  it  means  to  learn  x 

Lippman 

Education 

2003 

McPeck 

141.  Reflective  skepticism 

Lippman 

Education 

2003 

McPeck 

142.  'Critical  thinking  is  the  appropriate  use  of  reflective  scepticism  within  the  problem  area 
under  consideration.' 

Hare 

Education 

1999 

McPeck 

143.  Advanced  Analysis  includes  a  chapter  on  critical  thinking.  This  definition  comes  from  a 
psychology  book  David  G.  Myers.  Exploring  Psychology,  5th  ed.,  (New  York:  Worth,  2003. 
Definition  is:  An  intellectual  process  that  "examines  assumptions,  discerns  hidden  values, 
evaluates  evidence,  and  assesses  conclusions. 

Hall  &  Citrenbaum 

IA 

2010 

Myers 

144.  "The  ability  to  judge  the  plausibility  of  specific  assertions,  to  weigh  evidence,  to  assess 
the  logical  soundness  of  inferences,  to  construct  counter  arguments  and  alternative 
hypotheses" 

Analysis  101 

IA 

2008 

Nickerson, 
Perkins,  & 
Smith 
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145 

Thinking  appraisively  about  any  human  product,  whether  said,  made,  or  done 

Lippman 

Education 

2003 

No  source 

146 

Thinking  explicatively  and  interpretively,  as  critics  do 

Lippman 

Education 

2003 

No  source 

147 

The  application  of  theoretical  thinking  to  practical,  problematic  situations 

Lippman 

Education 

2003 

No  source 

148 

Reflections  upon  the  causes  and  consequences  of  what  happens 

Lippman 

Education 

2003 

No  source 

149 

The  reflective  assessment  of  practice 

Lippman 

Education 

2003 

No  source 

150 

Thinking  that  considers  how  to  facilitate  communication  between  experts  and  the  world 

Lippman 

Education 

2003 

No  source 

151 

A  systematic  search  for  the  reasons  by  which  one's  thinking  can  be  justified 

Lippman 

Education 

2003 

No  source 

152 

Thinking  that  seeks  to  disclose  the  persuasive  aspects  of  all  explanation,  while  holding 
that  all  explanation  is  argument 

Lippman 

Education 

2003 

No  source 

153 

Thinking  that  examines  differences  of  interpretation  resulting  from  differences  in 
contexts,  conceptual  schemes,  and  points  of  view 

Lippman 

Education 

2003 

No  source 

154 

A  light  version  of  philosophy 

Lippman 

Education 

2003 

No  source 

155 

The  testing  of  claims 

Lippman 

Education 

2003 

No  source 

156 

Literacy  (David  Olson) 

Lippman 

Education 

2003 

Olson 

157 

According  to  Pascarella  and  Terenzini  (1991,  p.  118),  "critical  thinking  appears  to  stress 
the  individual's  ability  to  interpret,  evaluate,  and  make  informed  judgments  about  the 
adequacy  of  arguments,  data,  and  conclusions." 

King  &  Kitchener 
Educational 
psychology 

1994 

Pascarella  and 
Terenzini 

158 

Thinking  that  contains  enabling  skills  (e.g.,  principles  of  reasoning,  skills  of  logic)  as 
well  as  skills  shared  across  fields  of  expertise  (Resnick) 

Lippman 

Education 

2003 

Resnick 

159 

Critical  thinking  is  the  process  of  examining  . . .  materials  in  the  light  of  related  objective 
evidence,  comparing  the  object  or  statement  with  some  norm  or  standard,  and  concluding  or 
acting  upon  the  judgment  then  made.  (Russell,  1956) 

Beyer 

Education 

1985 

Russell 
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160.  One  of  my  favorite  definitions  of  critical  thinking  was  published  over  40  years  ago 

(1960)  and  comes  very  close  to  this  contemporary  notion  of  critical  thinking  as  a  learnable 
skill:  "Critical  thinking  then  is  the  process  of  evaluation  or  categorization  in  terms  of  some 
previously  accepted  standards  .  .  .  this  seems  to  involve  attitude  plus  knowledge  of  facts  plus 
some  thinking  skills"  (Russell,  cited  in  d' Angelo,  1971,  p.  6).  In  short,  Russell's  equation  is: 

Attitude  +  Knowledge  +  Thinking  Skills  =  Intelligent  Thinking 

Halpem 

Psychology 

2003 

Russell 

161.  Nursing  specific  consensus  definition  (2000):  Critical  thinking  is  an  essential 
component  of  professional  accountability  and  quality  nursing  care.  Critical  thinkers  in 
nursing  exhibit  these  habits  of  the  mind:  confidence,  contextual  perspective,  creativity, 
flexibility,  inquisitiveness,  intellectual  integrity,  intuition,  open-mindedness,  perseverance, 
and  reflection.  Critical  thinkers  in  nursing  practice  the  cognitive  skills  of  analyzing, 
applying  standards,  discriminating,  information  seeking,  logical  reasoning,  predicting  and 
transforming  knowledge,  (p.  7) 

Staib 

Nursing 

2003 

Self 

162.  Introduction  to  the  Cambridge  Handbook  of  Thinking  and  Reasoning,  provides  a 
definition  for  Thinking  (general  rather  than  critical):  Thinking  is  the  systematic 
transformation  of  mental  representations  of  knowledge  to  characterize  actual  or  possible 
states  of  the  world,  often  in  service  of  goals. 

Holyoak  &  Morrison 
Psychology 

2005 

Self 

163.  LOGIC  can  be  briefly  defined  as  the  study  of  reasoning.  The  study  of  any  subject  calls 
for  thought,  and  every  student  is,  or  ought  to  be,  a  thinker;  but  he  is  not  a  student  of  logic 
unless  he  thinks  about  reasoning.  Now  reasoning  is  itself  a  special  kind  of  thinking;  hence 
the  special  kind  of  study  known  as  logic  is  concerned  with  thinking  about  thinking. 

Max  Black 
Philosophy 

1952 

Self 

164.  Definition  of  higher  order  thinking:  Students  engage  in  purposeful,  extended  lines  of 
thought  during  which  they: 

•  Identify  the  task  or  problem  type. 

•  Define  and  clarify  essential  elements  and  terms. 

•  Judge  and  connect  relevant  information. 

•  Evaluate  the  adequacy  of  information  and  procedures  for  drawing  conclusions  and/  or 
solving  problems. 

Quellmalz 

Education 

1985 

Self 

HOT 
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165.  Beyer’s  anti  definition:  The  term  critical  thinking  is  one  of  the  most  abused  terms  in 
our  thinking  skills  vocabulary.*  Generally  it  means  whatever  its  users  stipulate  it  to  mean. 

In  some  circles  the  term  critical  thinking  is  used  to  mean  all  thinking  operations,  from 
decision  making  to  analysis  of  part-whole  relationships  to  interpreting.  In  other  circles  it 
means  the  skills  drawn  from  Bloom's  taxonomy.  Y et  critical  thinking  is  not  to  be  considered 
as  encompassing  all,  or  identical  to  any,  of  these  operations.  Critical  thinking,  for  example, 
is  obviously  not  the  same  as  recall.  Neither  is  critical  thinking  synonymous  with  decision 
making  or  problem  solving. 

Beyer 

Education 

1987 

Self 

166.  Critical  thinking  has  been  defined  as  the  ability  "to  recognize  relevance,  dependability, 
bias  in  sources,  and  adequacy. 

Prawat 

Education 

1991 

Self 

167.  Critical  thinking  means  correct  thinking  in  the  pursuit  of  relevant  and  reliable 

knowledge  about  the  world. 

Schafersman 

Education 

1991 

Self 

168.  Critical  thinking  can  be  described  as  the  scientific  method  applied  by  ordinary  people  to 
the  ordinary  world. 

Schafersman 

Education 

1991 

Self 

169.  At  a  practical  level,  critical  thinking  is  effective,  adaptive  thinking 

Zechmeister  & 
Johnson 
Philosophy 

1992 

Self 

170.  Critical  thinking  is  an  important  attribute-the  ability  to  critique  the  sources  and  not 
assume  that  because  some  information  is  on  paper  and  was  acquired  clandestinely  it  must  be 
good. 

Graves 

IA 

1994 

Self 

171.  the  key  survival  skill  that  the  human  brain  must  master  without  help  from  computers. 

Fisher  &  Scriven 
Philosophy 

1997 

Self 

172.  a  'second-order  skill'  that  involves  the  intelligent  use  of  knowledge  and  reasoning  to 
weigh  the  value  of  claims  to  knowledge. 

Fisher  &  Scriven 
Philosophy 

1997 

Self 

173.  Critical  thinking  is  the  cutting  edge  of  thought 

Fisher  &  Scriven 
Philosophy 

1997 

Self 

174.  Critical  thinking  is  thinking  with  integrated  quality  control. 

Fisher  &  Scriven 
Philosophy 

1997 

Self 

175.  The  process  of  figuring  out  what  to  believe  or  not  about  a  situation,  phenomenon, 
problem  or  controversy  for  which  no  single  definitive  answer  or  solution  exists.  The  term 
implies  a  diligent,  open-minded  search  for  understanding,  rather  than  for  discovery  of  a 
necessary  conclusion. 

Mumm  &  Kersting 
Education 

1997 

Self 

153 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-2014-1598,  Cleared  11  April  2014 


Definition 

Author/Field 

Date 

Source 

176.  Critical  thinking  is  a  broader  term  describing  reasoning  in  an  open-ended  manner,  with 

an  unlimited  number  of  solutions.  The  critical  thinking  process  involves  constructing  the 
situation  and  supporting  the  reasoning  behind  a  solution. 

NCEC 

Education 

2000 

Self 

177.  Critical  thinking  involves  evaluating  a  situation,  problem,  or  argument  and  choosing  a 

path  of  investigation  that  leads  to  finding  the  best  possible  answers. 

Feldman 

Unknown 

2002 

Self 

178.  Critical  thinking  is  not  knowledge  about  a  problem  but  is  a  skill  of  examining  and 

thinking  about  a  problem.  It  begins  with  questioning  what  others  believe  to  be  fact  and 
realizing  that  there  is  more  than  one  way  of  examining  a  problem. 

Boba 

Law  enforcement 

2003 

Self 

179.  Oermann  (1997)  defined  critical  thinking  as  the  thought  process  that  underlies  effective 
clinical  problem  solving  and  decision  making. 

Staib 

Nursing 

2003 

Self 

180.  Alfaro-LeFevre  (1999)  described  critical  thinking  in  nursing  as  purposeful,  outcome 
directed  thinking  driven  by  patient  needs  and  based  on  principles  of  the  nursing  process  and 
scientific  method. 

Staib 

Nursing 

2003 

Self 

181.  Normative  analytic  methods  follow  the  general  principles  of  critical  thinking  that 
carefully  define  problems,  then  identify  assumptions,  decision  criteria,  evidence  and 
alternative  solutions  before  evaluating  the  effects  of  alterative  solutions  to  arrive  at  an 
optimal  decision. 

Waltz 

IA 

2003 

Self 

182.  Critical  thinking  is  based  on  the  application  of  a  systematic  method  to  guide  the 
collection  of  evidence,  reason  from  evidence  to  argument,  and  apply  objective  decision¬ 
making  judgment  (Table  4.10).  The  systematic  methodology  assures  completeness  (breadth 
of  consideration),  objectivity  (freedom  from  bias  in  sources,  evidence,  reasoning,  or 
judgment),  consistency  (repeatability  over  a  wide  range  of  problems),  and  rationality 
(consistency  with  logic).  In  addition,  critical  thinking  methodology  requires  the  explicit 
articulation  of  the  reasoning  process  to  allow  review  and  critique  by  others,  (p.  144) 

Waltz 

IA 

2003 

Self 

183.  critical  thinking  is  asking  and  answering  questions  about  alternative  possibilities  in 
order  to  achieve  some  objective.  Critical  thinking  is  (1)  a  question  and  answer  (or  more 
specifically,  a  challenge  and  defend)  dialogue  with  oneself  or  others  (2)  about  alternative 
possibilities  (3)  carried  out  for  a  purpose 

Cohen  et  al. 

Army 

2004 

Self 

184.  Critical  thinking  is  manifest  in  mindfulness,  investment  of  mental  effort,  and 

willingness  to  take  intellectual  risks. 

Florence  &  Yore 
Education 

2004 

Self 
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185.  A  more  concise  definition  of  critical  thinking  is:  the  ability  to  logically  assess  the 
quality  of  one’s  thinking  and  the  thinking  of  others  to  consistently  arrive  at  greater 
understanding  and  achieve  wise  judgments. 

Guillot 

Military 

2004 

Self 

186.  In  this  handbook,  “critical  thinking”  means  sound  thinking  needed  by  practitioners  in  an 
academic  discipline:  accurate,  relevant,  reasonable,  rigorous — whether  it  be  analyzing, 
synthesizing,  generalizing,  applying  concepts,  interpreting,  evaluating  supporting  arguments 
and  hypotheses,  solving  problems,  or  making  decisions. 

Prince  George 
Comm.  College 
Education 

2004 

Self 

187.  Indeed,  for  most  thinking  abilities  that  might  be  considered  naturally  occurring,  one  can 
usually  identify  a  more  sophisticated  form  that  such  thinking  might  take  with  some 
deliberate  nurturing.  This  type  of  thinking  is  what  is  often  referred  to  as  high-end  thinking 
or  critical  and  creative  thinking.  Such  thinking  extends  beyond  a  natural  processing  of  the 
world  into  the  realm  of  deliberative  thinking  acts  aimed  at  solving  problems,  making 
decisions  (see  LeBoeuf  &  Shafir,  Chap.  1 1),  and  forming  conclusions. 

Perkins  &  Ritchhart 
Education 

2005 

Self 

188.  critical  thinking  involves  examining  possibilities  carefully,  fairly,  and  constructively, 
and  then  focusing  your  thoughts  and  actions  by:  •  organizing  and  analyzing  possibilities,  • 
refining  and  developing  promising  possibilities,  •  ranking  or  prioritizing  options,  and  • 
choosing  or  deciding  on  certain  options. 

Treffinger,  Isaksen, 
Stead-Dorval 
Educational  psych 

2005 

Self 

189.  When  all  things  have  been  considered,  critical  thinking  is  found  to  make  three  important 
demands.  It  insists  on  clearness.  The  critical  thinker  is  always  asking  such  questions  as:  Just 
what  did  we  see?  Just  what  were  the  sounds  we  heard?  What  were  the  exact  words  used? 

What  did  he  really  mean  to  say?  What  did  he  do?  What  did  he  mean  to  do?  Critical  thinking 
makes  a  persistent  demand  for  all  the  actual  facts  in  the  case.  It  has  no  patience  with  merely 
alleged  facts.  It  rebels  against  all  reports  which  are  "made  to  order"  and  "fixed"  to  suit  the 
supposed  needs  of  the  hearer.  Finally,  critical  thinking  is  effective  thinking;  it  demands 
results.  It  is  opposed  to  speculations  which  lead  nowhere.  It  objects  to  everything  that  is 
irrelevant  and  immaterial.  To  some  people  critical  thinking  has  meant  merely  the  type  of 
thinking  that  demands  clearness  and  truth.  It  means  more  than  that.  It  insists  that  the 
particular  truth  which  we  hold  shall  be  one  that  has  value  in  reaching  a  conclusion. 

Boraas 

Education 

1922 

Self 

190.  At  Frostburg  State  University,  critical  thinking  is  defined  as  reflective,  self-directed 
thinking  that  requires  skills  in  reasoning  and  the  dispositions  to  use  those  skills  so  that  a 
person  can  think  effectively  about  questions,  problems,  and  decisions  both  inside  and 
outside  of  the  classroom. 

Bensley  et  al. 
Education 

2006 

Self 

With  some 
influence  from 
Dewey  &  Ennis 
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191.  Critical  thinking  is  a  term  used  to  refer  to  those  kinds  of  mental  activity  that  are  clear, 
precise  and  purposeful.  It  is  typically  associated  with  solving  complex  real-world  problems, 
generating  multiple  -  or  creative  -  solutions  to  a  problem,  drawing  inferences,  synthesising 
and  integrating  information,  distinguishing  between  fact  and  opinion,  or  estimating  potential 
outcomes,  but  it  can  also  refer  to  the  process  of  evaluating  the  quality  of  one’s  own  thinking. 

Hart  &  Simon 

IA 

2006 

Self 

Ref  to 

foundation  also 

192.  Critical  thinking  consists  of  seeing  both  sides  of  an  issue,  being  open  to  new  evidence 
that  disconfirms  your  ideas,  reasoning  dispassionately,  demanding  that  claims  be  backed  by 
evidence,  deducing  and  inferring  conclusions  from  available  facts,  solving  problems,  and  so 
forth. 

Willingham 

Education 

2007 

Self 

193.  Critical  thinking  is  quality  control  for  problem  solving  and  reasoning.  Critical  thinking 

enables  you  to  judge  the  accuracy  and  relevance  of  your  information,  categorize  information 
as  assumption,  fact,  or  inference,  identify  gaps  in  your  knowledge,  recognize  factors  that 
influence  your  objectivity  and  the  objectivity  of  others,  and  judge  the  validity  of  your 
arguments. 

Analysis  101 

IA 

2008 

Self 

194.  Intentionally  applying  rigorous  analytic  procedures  to  relevant  analyst  problems  for 
reliable  analytic  products.  Each  approach  to  critical  thinking  has,  then,  three  dimensions: 
procedures,  problems,  and  products.  The  procedures  are  methods  of  reasoning  that  the 
approach  offers.  The  problems  are  the  challenges  that  the  procedures  are  designed  to 
address.  And,  the  products  are  the  results  of  applying  the  procedures  to  the  problems.  So,  the 
approaches  can  be  distinguished  in  terms  of  differences  in  their  procedures,  problems,  and 
products. 

Hendrickson 

IA 

2008 

Self 

195.  Critical  thinking,  therefore,  may  be  defined  as  the  process  by  which  we  test  claims  and 
arguments  and  determine  which  have  merit  and  which  do  not.  In  other  words,  critical 
thinking  is  a  search  for  answers,  a  quest. 

Ruggiero 

Education 

2008 

Self 

196.  Critical  thinking  involves  analyzing  a  problem  in  depth,  which  in  turn  often  requires 

extrapolation  and  the  projection  of  observed  patterns  and  trends  into  the  long-  or  short-term 
future  based  on  statistics  and  mathematical  probability.  Such  trend  projection  is  typical  in 
the  related  group  of  disciplines  known  as  neoclassical  economics,  operations  research, 
management  science,  and  decision  theory.  It  relies  on  strict  deductive  inference  from  given 
premises,  including  the  precept  that  man  is  a  utility-maximizing  "rational  actor"  ("economic 
man"),  (p.  270) 

Smith 

IA 

2008 

Self 
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197.  definition  for  higher  order  thinking  which  includes  both  critical  and  creative  thinking: 

Higher-order  thinking  is  the  mental  engagement  with  ideas,  objects,  and  situations  in  an 
analogical,  elaborative,  inductive,  deductive,  and  otherwise  transformational  manner  that  is 
indicative  of  an  orientation  toward  knowing  as  a  complex,  effortful,  generative,  evidence¬ 
seeking,  and  reflective  enterprise. 

Alexander  et  al. 
Education 

2009 

Self 

HOT 

198.  critical  thinking  is  the  careful  application  of  reason  in  the  determination  of  whether  a 

claim  is  true. 

Moore  &  Parker 
Philosophy 

2009 

Self 

199.  We  also  developed  a  series  of  shorthand  labels  such  as  "quality  control  of  the  mind." 

Eichhorn 

Army 

2010 

Self 

200.  Critical  thinking  is  an  organized  and  disciplined  way  of  thinking.  It  is  thinking  logically 
with  clarity  and  precision;  using  an  approach  that  is  fair  and  accurate;  focusing  on 
information  that  is  relevant.  Also  introduced  a  new  concept: 

Applied  critical  thinking  is:  Taking  a  fair-minded  approach  to  thinking  logically  and 
comprehensively  about  a  particular  situation,  then  identifying  an  appropriate  course  of 
action 

Ishikawa 

Business 

Unknow 

n 

(recent) 

Self 

20 1 .  Critical  Thinking  is  the  ability  to  logically  and  comprehensively  evaluate  a  situation  and 
identify  the  most  appropriate  course  of  action. 

Action  Mgt  Assoc 
Business 

Undated 

Self 

202.  In  this  paper,  critical  thinking  is  regarded  as  the  set  of  intellectual  processes  or 

operations  that  occur  when  people  respond  critically  to  instances  of  valid  and  invalid 
reasoning,  argument  or  evidence.  Major  elements  of  CT  are  (1)  the  process  of  discriminating 
between  valid  and  invalid  instances,  on  the  basis  of  acceptable  criteria,  (2)  the  process  of 
identifying  defects  in  invalid  instances  on  the  basis  of  acceptable  criteria,  and  (3)  the  process 
of  responding  critically  to  invalid  instances. 

Wright 

Education 

1977 

Self 

203.  Although  managers  did  not  elaborate  on  what  they  meant  by  critical  thinking,  this  term 
generally  refers  to  a  process  of  purposeful  thinking  about  a  subject  that  employs  logic, 
examination  of  the  evidence  and  assumptions,  and  evaluation  of  the  thinking  process  itself. 

Derbentseva, 
McLellan,  Mandel, 
IA 

2010 

Self 

204.  Critical  thinking  is  the  disciplined  mental  activity  of  evaluating  arguments  or 

propositions  and  making  judgments  that  can  guide  the  development  of  beliefs  and  taking 
action. 

Huitt 

Psychology 

1998 

Self,  but  based 
on  Ennis 

205.  "Construction  of  a  logical  argument  based  on  rigorous  weighing  of  evidence  and 
assumptions,  and  the  assessment  of  multiple  hypotheses,  resulting  in  accurate,  persuasive, 
and  policy-relevant  conclusions" 

Analysis  101 

IA 

2008 

Sherman  Kent 
School 
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206.  "reasons"  conception  of  critical  thinking,  according  to  which  a  critical  thinker  is  one 

who  is  appropriately  moved  by  reasons. 

Siegel 

Education  & 
philosophy 

1988 

Siegel 

207.  Siegel  11988)  notes  "a  critical  thinker  is  one  who  is  appropriately  moved  by  reasons:  she 
has  a  propensity  and  disposition  to  believe  and  act  in  accordance  with  reasons;  she  has  the 
ability  to  assess  the  force  of  reasons  in  the  many  contexts  in  which  reasons  play  a  role"  (p. 

23).  Siegel's  theory  asserts  a  close  connection  between  critical  thinking,  rationality,  and 
problem  solving. 

Jones  et  al. 
Education 

1995 

Siegel 

208.  For  Siegel  the  reason-assessment  component  is  central  to  critical  thinking  and  involves 
the  ability  properly  to  assess  reasons  and  their  ability  to  warrant  beliefs,  claims  and  actions 

Bailin 

Education 

1997 

Siegel 

209.  Siegel's  definition  of  critical  thinking  is  thinking  that  is  "appropriately  moved  be 
reasons" 

Hatcher 

Education/philosophy 

2000 

Siegel 

210.  Thinking  performed  by  those  who  are  appropriately  moved  by  reasons  (Siegel) 

Lippman 

Education 

2003 

Siegel 

211.  From  Siegel:  . .  .being  a  critical  thinker  requires  basing  one’s  beliefs  and  actions  on 
reasons. . .  the  beliefs  and  actions  of  the  critical  thinker,  at  least  ideally,  are  justified  by 
reasons  for  them  which  she  has  properly  evaluated  [italics  in  original]. 

Cohen  et  al. 

Army 

2004 

Siegel 

212.  'Critical  thinking  is  thinking  which  appropriately  reflects  the  power  and  convicting  force 
of  reasons.'  (Siegel) 

Hare 

Education 

1999 

Siegel 

213.  'Critical  thinking  is  the  conscious,  deliberate  rational  assessment  of  claims  according  to 

clearly  identified  standards  of  proof.'  (Soccio) 

Hare 

Education 

1999 

Soccio 

214.  Construed  broadly,  critical  thinking  comprises  the  mental  processes,  strategies,  and 
representations  people  use  to  solve  problems,  make  decisions,  and  learn  new  concepts. 

Sternberg 

Psychology 

1986 

Sternberg 

215.  Thinking  that  helps  us  solve  problems  and  make  decisions  (Sternberg) 

Lippman 

Education 

2003 

Sternberg 

216.  In  a  recent  review  of  the  critical  thinking  literature,  Fischer  and  Spiker  (2000)  found  that 
most  definitions  for  the  term  "critical  thinking"  include  reasoning/logic,  judgment, 
metacognition,  reflection,  questioning,  and  mental  processes. 

Halpem 

Psychology 

2003 

summary 

217.  definition  of  thinking  -  not  of  CT:  “That  brings  us  close  to  the  definition  of  thinking  we 
promised  earlier.  What  is  it  that  really  goes  on  in  the  mind?  To  us,  it's  nothing  less  than  the 
making  of  meaning.  And  we  mean  making  in  its  most  literal  sense — as  in  "making  sense" 
and  "making  up  our  minds."  We're  convinced  that  every  mind  creates  its  own  understanding 
by  pulling  things  together  in  its  own  way,  and  "thinking"  is  what  we  call  that  process.” 

Kirby  and 
Kuykendall 
Education 

1991 

Thinking  in 
general,  not  CT 
Self  generated. 

158 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-2014-1598,  Cleared  11  April  2014 


Definition 

Author/Field 

Date 

Source 

218.  from  Walters:  Walters  holds  that  there  is  a  more  holistic  view  of  rationality,  one  that 
includes,  along  with  the  logical  processes,  intuition,  imagination,  conceptual  creativity,  and 
insight.  He  argues  that  much  of  the  bandwagon  effect  of  critical  thinking  assumes  that 
critical  thinking  is  logical  thinking.  Although  Walters  agrees  (as  do  I)  that  logical  inference, 
critical  analysis,  and  problem  solving  are  fundamental  elements  of  good  thinking,  they  are 
practically  useful  only  if  they  are  supplemented  by  imagination,  insight,  and  intuition,  which 
he  considers  essential  components  of  discovery. 

Jonassen 

Education 

2000 

Walters 
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LIST  OF  ACRONYMS 


ACH 

Analysis  of  Competing  Hypotheses 

ACT 

American  College  Test 

APA 

American  Philosophical  Association 

ATIC 

Advanced  Technical  Intelligence  Center 

CCTDI 

California  Critical  Thinking  Dispositions  Inventory 

CCTST 

California  Critical  Thinking  Skills  Test 

CIA 

Central  Intelligence  Agency 

CLA 

Collegiate  Learning  Assessment 

COCOM 

Combatant  Command 

CT 

Critical  Thinking 

CTSA 

Critical  Thinking  and  Structured  Analysis 

DIA 

Defense  Intelligence  Agency 

DNI 

Director  of  National  Intelligence 

FAC 

Framework  for  Analytic  Cognition 

FIA 

Fundamentals  of  Intelligence  Analysis 

HECB 

Higher  Education  Coordinating  Board 

HCTA 

Halpern  Critical  Thinking  Assessment 

IC 

Intelligence  Community 

ICD 

Intelligence  Community  Directive 

IQ 

Intelligence  quotient 

ISA 

International  Studies  Association 

LAMP 

Lots  of  Argument  Mapping  Practice 

NASIC 

National  Air  and  Space  Intelligence  Center 

NIE 

National  Intelligence  Estimate 

NIPF 

National  Intelligence  Priorities  Framework 

ODNI 

Office  of  the  Director  of  National  Intelligence 

QPH 

Quality  Practice  Hypothesis 

SAT 

Scholastic  Aptitude  Test 

SD 

School  District 

TIACRITIS 

Teaching  Intelligence  Analysts  Critical  Thinking 
Skills 

WGCTA 

Watson-Glaser  Critical  Thinking  Assessment 

WSU 

Washington  State  University 
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